Custom Logs in Azure with Data Collection Rules and Endpoints

Introduction

Centralized log collection is a key component of modern cloud architectures. It’s no longer just about classic error analysis, but also about security monitoring, compliance reporting, automated alerting, and the continuous improvement of applications.

In the past, open-source tools from the Elastic or Grafana suite were often used to collect logs and metrics from workloads. However, these solutions add operational complexity, offer limited integration with Azure resources, and lack proper governance integration.

Azure offers a native alternative with Log Analytics Workspace, fully embedded within the Azure platform. This architecture enables the secure, scalable, and cost-efficient transmission of structured log data from any source, including applications, CI/CD pipelines, or third-party systems.

At the same time, governance mechanisms, network isolation, RBAC, transformation, and automation are supported as integral components.

In this post, I’ll take a detailed look at Data Collection Endpoints (DCE) and Data Collection Rules (DCR). These Azure components enable the creation and storage of custom log files in a user-defined schema. With them, we can extend Azure's already extensive logging capabilities to meet our specific requirements.

Solution Architecture

The solution is based on a modular, Azure-native ingestion pipeline, with the Log Analytics Workspace (LAW) at its core. LAW serves as a scalable, Kusto-based analytics platform that stores both structured and semi-structured log data in custom tables.

The LAW functions not only as the persistent destination but also as the foundation for queries, visualizations, correlations, and automated responses to log events.

External or internal log sources are connected via the Logs Ingestion API, which is accessed over HTTPS. Each incoming log message must be in a structured JSON format and is associated with a Data Collection Rule (DCR). The rule defines the expected schema of the data and what fields are expected, their data types and the target table in LAW. Additionally, DCRs can include optional Kusto transformations that are applied during ingestion. These allow you to compute, enrich, rename, or filter fields before the data is persistently stored, which can significantly reduce both costs and storage usage.

The data transmission is initiated by your client for example, an application or a CI/CD pipeline, which prepares the log data and authenticates to Azure via Microsoft Entra ID. Typically, this uses a client credentials flow, although using a managed identity is also possible for Azure-native resources. Ingestion requests always reference an immutable DCR ID, which is tightly coupled to the rule and its associated data stream.

Ingestion can occur either directly via the DCR or via a preceding Data Collection Endpoint (DCE). The latter is especially necessary when network isolation using Private Link is required or when certain platform requirements mandate it. The DCE provides a dedicated, regionally distributed ingestion URL and also acts as a management endpoint for configuration and authentication.

Once the upload reaches the DCE or DCR, Azure Monitor takes over. The JSON body is validated against the declared schema, optionally transformed, and finally persisted to the appropriate table in LAW. This happens with low latency and is fully managed by Azure, including scaling, load balancing, and retention management.

A key feature of this architecture is transformation on ingest. By manipulating data early, you can discard or normalize irrelevant information (such as standardizing log levels) and even derive metrics such as latency classes, error codes, or anomalies before persistence. This not only reduces downstream analysis efforts but also optimizes the cost structure, since Azure Monitor charges are primarily based on ingestion volume and query load.

This architecture enables decoupled, scalable, and secure log collection across system and platform boundaries with full control over format, access, and processing.

Log Analytics Workspace

The implementation always starts with the creation of a Log Analytics Workspace (LAW), which serves as the central platform in Azure for storing, analyzing, and correlating log and metric data. You can create it via the Azure Portal, CLI, or Infrastructure-as-Code using Terraform or Bicep. Naming conventions, tags, and resource group structures should be planned early to meet existing governance requirements.

One of LAW’s key advantages is its diversity of data sources. Logs can be aggregated from native Azure services, on-premises systems using the Azure Monitor Agent, or even from multi-cloud environments.

For custom logs, a custom table like Custom_Table_1_CL is often used. Such tables require at least a timestamp field named TimeGenerated and a payload field,typically RawData or Message. Additional columns can be dynamically created at ingest through transformations defined in the Data Collection Rule using KQL for example, by extracting individual fields, renaming them, or converting data types. This decoupling of raw data from the table schema enables high flexibility in processing and analysis.

The Log Analytics platform provides significant added value. In addition to storing logs, it enables interactive analysis using the Kusto Query Language (KQL), the creation of custom dashboards and workbooks, and the definition of rule-based alerts. These alerts can respond to virtually any KQL query and trigger automated actions, such as sending notifications or running Logic Apps.

Integration with other Azure services is deep and often available out of the box. Resources like Azure App Services, Application Gateway, or Key Vault can send logs and metrics directly to LAW via Diagnostic Settings, without requiring agents or manual configuration. Additionally, the Logs Ingestion API allows for integration with any application or service capable of sending HTTP requests ideal for custom applications, CI/CD pipelines, or external systems.

In summary, the Log Analytics Workspace is not just the endpoint for log ingestion, it serves as the heart of telemetry in the Azure Cloud. It unifies data flows, enforces centralized security and access policies, scales automatically with data volume, and lays the foundation for modern, automated operations in cloud and hybrid environments.

Data Collection Endpoint

In scenarios with heightened security requirements, a Data Collection Endpoint (DCE) is set up in addition to the Log Analytics Workspace. The DCE provides a dedicated ingestion URL that receives structured log data and routes it to the associated Data Collection Rule (DCR). Unlike direct DCR ingestion, a DCE gives you full control over network communication between the client and Azure Monitor, especially through support for Azure Private Link, Network Security Groups (NSGs), and subnet-level firewall rules.

A DCE is deployed regionally and is tied to a specific Azure region. After deployment, it provides a dedicated endpoint in the format: https://(dce-name-placeholder).(region-placeholder).ingest.monitor.azure.com, which acts as the target for all log ingestion requests. Clients send their POST requests with log data to this endpoint, while still explicitly referencing the target DCR by ID.

Integrating the DCE into a Virtual Network (VNet) allows you to restrict incoming ingestion requests to trusted sources only. Combined with NSGs and Private Link, the entire data path can be fully isolated from the public internet. This is essential in environments with strict compliance and data protection requirements, such as in regulated industries or public sector organizations.

Additionally, a DCE can be used to address multiple DCRs in a consolidated manner, simplifying operations in larger organizations or tenant structures. Separating ingestion (DCE) from data processing (DCR) results in clear responsibilities and better reusability of configurations.

While a DCE is technically optional, especially now that DCRs can serve as direct endpoints, it remains essential in production environments. Once Private Link is activated, a DCE is required. It is also strongly recommended for zero-trust architectures, where all incoming connections to Azure Monitor must be explicitly controlled.

In short, the Data Collection Endpoint serves as the central bridge between internal log sources and the Azure Monitor infrastructure, offering full support for network segmentation, security policies, and tenant-scalable designs.

Data Collection Rule

The Data Collection Rule (DCR) forms the logical configuration layer within the Azure Monitor ingestion architecture. It defines how incoming log data is to be processed including schema, transformations, and the target resource such as a custom table in Log Analytics Workspace.

DCRs are typically defined in JSON format and consist of three key sections:

• Declaration of expected data streams
• Optional Kusto transformations
• Definition of the output destination

In the streamDeclarations, expected fields, their types, and mapping to a named data stream are defined such as Custom-DemoApp, which maps to a table like DemoApp_CL.

The most powerful part of a DCR is the transformKql section, where Kusto Query Language (KQL) is used to manipulate incoming data before it’s stored. You can derive numeric severity levels from text fields, split unstructured fields into dimensions using parse_json() or extract(), or selectively filter out sensitive content such as IP addresses or user IDs. Transformations at the ingestion level are not only useful for improving data quality but also for optimizing costs by ensuring irrelevant or redundant data is never stored.

In projects with heterogeneous or dynamic log sources, it may be necessary to preprocess data upstream e.g., using custom scripts or agents that convert data to the expected JSON format before upload, eliminate duplicates, or create aggregated payloads. This preprocessing complements the KQL transformations and enables two-stage control over data quality.

Using a DCR requires secure authentication. Azure requires a valid OAuth2 token, issued by Microsoft Entra ID, to access the Logs Ingestion API. This is typically done using either a Service Principal (with client credentials flow) or a system/user-assigned Managed Identity. The minimum required permission is the RBAC role “Monitoring Metrics Publisher”, assigned specifically to the DCR. The token must be requested for the resource https://monitor.azure.com and passed in the Authorization header as a Bearer token with every API call.

With this separation of authentication, transformation, and destination mapping, the Data Collection Rule becomes the central control mechanism for log ingestion, flexible enough for simple log structures, yet powerful enough to handle complex enterprise data streams.

Technical Implementation with Terraform

Below you'll find a compact template for creating the Data Collection Endpoints, Rules, and all associated resources using Terraform. Note: For readability, I’ve deliberately omitted module structures, variables, and extensibility features. Please adapt the template to your specific project requirements.

[...]
# ---------------------------------------------------------------------------
# Log Analytics Workspace (LAW)
# ---------------------------------------------------------------------------

resource "azurerm_resource_group" "law_rg" {
  name     = "rg-law-prod"
  location = "westeurope"

  tags = {
    environment = "prod"
    project     = "example"
  }
}

resource "azurerm_log_analytics_workspace" "law" {
  name                       = "law-prod-01"
  location                   = azurerm_resource_group.law_rg.location
  resource_group_name        = azurerm_resource_group.law_rg.name
  sku                        = "PerGB2018"
  retention_in_days          = 30
  internet_ingestion_enabled = false
  internet_query_enabled     = true

  tags = {
    environment = "prod"
    project     = "example"
  }
}

# ---------------------------------------------------------------------------
# Azure Monitor Private Link Scope (AMPLS)
# ---------------------------------------------------------------------------

resource "azurerm_monitor_private_link_scope" "ampls" {
  name                = "ampls-law-prod"
  resource_group_name = azurerm_resource_group.law_rg.name
  location            = azurerm_resource_group.law_rg.location

  ingestion_access_mode = "PrivateOnly"
  query_access_mode     = "Open"

  tags = {
    environment = "prod"
    project     = "example"
  }
}

resource "azurerm_monitor_private_link_scoped_service" "ampls_linked_law" {
  name                = "ampls-law-prod-law"
  resource_group_name = azurerm_resource_group.law_rg.name
  scope_name          = azurerm_monitor_private_link_scope.ampls.name
  linked_resource_id  = azurerm_log_analytics_workspace.law.id
}

# ---------------------------------------------------------------------------
# Private Endpoint zum AMPLS
# ---------------------------------------------------------------------------

resource "azurerm_private_endpoint" "ampls_pe" {
  name                = "pe-law-prod-01"
  location            = azurerm_resource_group.law_rg.location
  resource_group_name = azurerm_resource_group.law_rg.name
  subnet_id           = data.azurerm_subnet.pe_subnet.id

  private_service_connection {
    name                           = "law-prod-01-pe-ampls"
    private_connection_resource_id = azurerm_monitor_private_link_scope.ampls.id
    subresource_names              = ["azuremonitor"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [
      data.azurerm_private_dns_zone.monitor.id,
      data.azurerm_private_dns_zone.omsopeninsights.id,
      data.azurerm_private_dns_zone.odsopeninsights.id,
      data.azurerm_private_dns_zone.agentsvc.id,
      data.azurerm_private_dns_zone.blobcore.id,
    ]
  }

  tags = {
    environment = "prod"
    project     = "example"
  }
}

###############################################################################
# Data‑Collection‑Endpoint (DCE)
###############################################################################

resource "azurerm_monitor_data_collection_endpoint" "dce" {
  name                          = "dce-law-prod"
  resource_group_name           = azurerm_resource_group.law_rg.name
  location                      = azurerm_resource_group.law_rg.location
  kind                          = "Linux"
  public_network_access_enabled = false

  tags = {
    environment = "prod"
    project     = "example"
  }
}

###############################################################################
# Custom Log Table (azapi) + Data‑Collection‑Rule (DCR)
###############################################################################

# Example definition: 1 custom log source named "CustomLogExample".
# To add more sources, simply duplicate this block and change the name.

locals {
  custom_log_name              = "CustomLogExample"
  custom_log_table_name        = "CustomLogExample_CL"
  custom_log_retention_days    = 30
  custom_log_total_retention   = 365
}

resource "azapi_resource" "law_customlog_table" {
  name      = local.custom_log_table_name
  parent_id = azurerm_log_analytics_workspace.law.id
  type      = "Microsoft.OperationalInsights/workspaces/tables@2022-10-01"

  body = jsonencode({
    properties = {
      schema = {
        name    = local.custom_log_table_name
        columns = [
          { name = "TimeGenerated", type = "datetime" },
          { name = "Message",       type = "string"   }
        ]
      }
      retentionInDays      = local.custom_log_retention_days
      totalRetentionInDays = local.custom_log_total_retention
    }
  })
}

resource "azurerm_monitor_data_collection_rule" "law_dcr_customlog" {
  name                        = "${local.custom_log_name}DataCollectionRule"
  resource_group_name         = azurerm_resource_group.law_rg.name
  location                    = azurerm_resource_group.law_rg.location
  data_collection_endpoint_id = azurerm_monitor_data_collection_endpoint.dce.id

  destinations {
    log_analytics {
      name                  = azurerm_log_analytics_workspace.law.name
      workspace_resource_id = azurerm_log_analytics_workspace.law.id
    }
  }

  data_flow {
    streams       = ["Custom-${azapi_resource.law_customlog_table.name}"]
    destinations  = [azurerm_log_analytics_workspace.law.name]
    output_stream = "Custom-${azapi_resource.law_customlog_table.name}"
  }

  stream_declaration {
    stream_name = "Custom-${azapi_resource.law_customlog_table.name}"

    column {
      name = "TimeGenerated"
      type = "datetime"
    }

    column {
      name = "Message"
      type = "string"
    }
  }

  tags = {
    environment = "prod"
    project     = "example"
  }
}

Log Ingestion Using a Bash Script

To ensure that your newly created Data Collection Rules actually receive data, you need a simple way to upload local or pipeline-generated JSON files to the Azure Monitor Ingestion API endpoint.

Below is a generic Bash script that follows key best practices:

#!/usr/bin/env bash
set -euo pipefail

###############################################################################
# Configuration                                                             #
###############################################################################
SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
TENANT_ID="00000000-0000-0000-0000-000000000000"

# Azure Monitor Ingestion
RESOURCE="https://monitor.azure.com"
DCR_ID="dcr-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
TABLE_NAME="MyCustomLog_CL"

# Region-Segment aus der DCE-URL, z. B. "westeurope-1"
REGION_SEGMENT="westeurope-1"

# endgültiger Ingestion-Endpoint
DCR_ENDPOINT="https://mdce-${REGION_SEGMENT}.ingest.monitor.azure.com/dataCollectionRules/
                ${DCR_ID}/streams/Custom-${TABLE_NAME}?api-version=2023-01-01"

# Sekunden Verzögerung zwischen zwei Uploads
SLEEP_SECONDS=1

###############################################################################
# Login + Access Token                                                      #
###############################################################################
echo "Authenticating via managed identity …"
az login --identity --tenant "$TENANT_ID" >/dev/null

az account set --subscription "$SUBSCRIPTION_ID"

BEARER_TOKEN="$(az account get-access-token \
  --resource "$RESOURCE" \
  --query accessToken -o tsv)"

###############################################################################
# Helper function: Curl with simple retry mechanism                         #
###############################################################################
retry() {
  local n=1
  local max=5
  local delay=2
  while true; do
    "$@" && break || {
      if [[ $n -lt $max ]]; then
        ((n++))
        echo "Retry $n/$max …"
        sleep $delay
      else
        echo "Error: Endpoint unreachable – aborting." >&2
        return 1
      fi
    }
  done
}

###############################################################################
# Upload loop                                                               #
###############################################################################
for file in *.json; do
  # Skip empty arrays or zero-byte files
  if [[ ! -s "$file" ]] || grep -q '^[[:space:]]*\\[\\][[:space:]]*$' "$file"; then
    echo "Skip: $file contains no usable data"
    continue
  fi

  echo "Uploading $file …"
  status_code=$(retry curl -s -o /dev/null -w "%{http_code}" -X POST "$DCR_ENDPOINT" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $BEARER_TOKEN" \
    --data-binary @"$file")

  if [[ "$status_code" == "204" ]]; then
    echo "✔ Upload successful (HTTP 204)"
  else
    echo "✖ Upload failed (HTTP $status_code)" >&2
  fi

  sleep "$SLEEP_SECONDS"
done

Advantages of This Solution

Disadvantages of This Solution

Summary

Compared to the classic Grafana Pushgateway approach, the Azure-native solution is clearly superior. While Pushgateway lacks robust authentication or schema validation, Azure enforces strict structuring of data streams right at the point of entry.

Transformations happen early, before persistence and not later in dashboards or query layers.

This improves:

• Performance of queries, since logs are already normalized.
• Data integrity, as malformed payloads are rejected immediately.
• Security & Governance (RBAC, Private Link, no exposed port 9091)

In short, we accept a bit more initial setup effort in exchange for lower long-term operating and maintenance costs and better auditability.

Helpful Links for Further Research

• Overview of the Ingestion API, including authentication, limitations, and payload format: Logs Ingestion API – Azure Monitor (offizielle Dokumentation)

• Fundamentals of DCR, including structure, use cases, and relationship to DCE and Log Analytics: Data Collection Rules (DCR) – Azure Monitor

• Explanation of when a DCE is needed, how regions work, and how the ingestion URL is composed: Data Collection Endpoints (DCE) – Azure Monitor

• AMPLS security architecture, including examples of DNS zones, endpoint integration, and restrictions: Private Link für Azure Monitor und Log Analytics (AMPLS)