I wanted to convert my Nginx logs to JSON and was hoping to utilize DataDog's built-in pipelines and parsers to ingest them without heavy (or any) customization. Indeed DataDog even wrote a guide on how to do it! Unfortunately the guide sent me in the completely wrong direction since it gives an nginx.conf log format which DataDog's own systems will not natively make searchable. Sure it will parse the JSON into a pretty tree to display, but not much more than that. This guide from Zendesk Engineering was much more useful.
It took a lot of trial and error but I think I finally got an Nginx log_format configuration that DataDog will natively ingest requiring no customizations in their dashboard. Customizing the ingestion is powerful but it's fragile and then you forget you ever did it when something breaks or changes in the future.
To enable this we need to name our log fields following DataDog's Standard Attributes guide. We can still add custom values, such as I did for x_forwarded_for, and create a facet in the DataDog dashboard to be able to filter on those custom values. I just wanted to avoid as much of that as possible.
Another feature I wanted was millisecond precision directly from the timestamp in the Nginx log. By default DataDog displays its own timestamp appended by the datadog-agent running on the server which can differ from the logged value. However neither the $time_local or $time_iso8601 variables of Nginx include milliseconds, but $msec does!
Alas $msec emits a 10.3 epoch.millisecond format, while DataDog only supports the 13 digit millisecond epoch format. I combined a couple solutions I found on online to create a map which concats the 10.3 format into the 13 digit format we need.
Putting all the pieces together, we end up with:
http {
...
map $msec $msec_no_decimal { ~(.*)\.(.*) $1$2; }
log_format json_datadog escape=json
'{'
'"timestamp":"$msec_no_decimal",'
'"http":{'
'"method":"$request_method",'
'"request_id":"$request_id",'
'"status_code":$status,'
'"content_type":"$content_type",'
'"useragent":"$http_user_agent",'
'"referrer":"$http_referer",'
'"x_forwarded_for":"$http_x_forwarded_for",'
'"url":"$request_uri"'
'},'
'"network":{'
'"bytes_written":$bytes_sent,'
'"bytes_read":$request_length,'
'"client":{'
'"ip":"$remote_addr",'
'"port":$remote_port'
'},'
'"destination":{'
'"ip":"$server_addr",'
'"port":$server_port'
'},'
'"nginx":{'
'"request_time":$request_time,'
'"upstream_connect_time":$upstream_connect_time,'
'"upstream_response_time":$upstream_response_time,'
'"upstream_header_time":$upstream_header_time'
'}'
'}'
'}';
...
}
server {
...
access_log /var/log/nginx/radsite-access.log json_datadog;
...
}
Which creates output in the DataDog dashboard like:
{
"http":{
"status_code":200,
"status_category":"OK",
"content_type":"application/json",
"referrer":"",
"url":"/some/path?user_id=20k40ffk",
"url_details":{
"path":"/some/path",
"queryString":{
"user_id":"20k40ffk"
}
},
"method":"GET",
"request_id":"d4f70c20f1c9cf8263753e601c0f3594",
"useragent":"CFNetwork/976 Darwin/18.2.0",
"useragent_details":{
"os":{
"family":"iOS",
"major":"12"
},
"browser":{
"family":"safari",
"major":"3"
},
"device":{
"family":"iOS-Device",
"model":"iOS-Device",
"category":"Mobile",
"brand":"Apple"
}
},
"x_forwarded_for":"1.1.2.2"
},
"network":{
"bytes_written":928,
"bytes_read":879,
"client":{
"ip":"10.10.10.2",
"port":30020
},
"destination":{
"port":443,
"ip":"10.10.10.3"
},
"nginx":{
"request_time":0.025,
"upstream_connect_time":0.004,
"upstream_header_time":0.024,
"upstream_response_time":0.024
}
},
"timestamp":"1571897461960"
}