I wanted to convert my Nginx logs to JSON and was hoping to utilize DataDog's built-in pipelines and parsers to ingest them without heavy (or any) customization. Indeed DataDog even wrote a guide on how to do it! Unfortunately the guide sent me in the completely wrong direction since it gives an nginx.conf log format which DataDog's own systems will not natively make searchable. Sure it will parse the JSON into a pretty tree to display, but not much more than that. This guide from Zendesk Engineering was much more useful.

It took a lot of trial and error but I think I finally got an Nginx log_format configuration that DataDog will natively ingest requiring no customizations in their dashboard. Customizing the ingestion is powerful but it's fragile and then you forget you ever did it when something breaks or changes in the future.

To enable this we need to name our log fields following DataDog's Standard Attributes guide. We can still add custom values, such as I did for x_forwarded_for, and create a facet in the DataDog dashboard to be able to filter on those custom values. I just wanted to avoid as much of that as possible.

Another feature I wanted was millisecond precision directly from the timestamp in the Nginx log. By default DataDog displays its own timestamp appended by the datadog-agent running on the server which can differ from the logged value. However neither the $time_local or $time_iso8601 variables of Nginx include milliseconds, but $msec does!

Alas $msec emits a 10.3 epoch.millisecond format, while DataDog only supports the 13 digit millisecond epoch format. I combined a couple solutions I found on online to create a map which concats the 10.3 format into the 13 digit format we need.

Putting all the pieces together, we end up with:

http {
  ...

  map $msec $msec_no_decimal { ~(.*)\.(.*) $1$2; }

  log_format json_datadog escape=json
  '{'
    '"timestamp":"$msec_no_decimal",'
    '"http":{'
      '"method":"$request_method",'
      '"request_id":"$request_id",'
      '"status_code":$status,'
      '"content_type":"$content_type",'
      '"useragent":"$http_user_agent",'
      '"referrer":"$http_referer",'
      '"x_forwarded_for":"$http_x_forwarded_for",'
      '"url":"$request_uri"'
    '},'
    '"network":{'
      '"bytes_written":$bytes_sent,'
      '"bytes_read":$request_length,'
      '"client":{'
        '"ip":"$remote_addr",'
        '"port":$remote_port'
      '},'
      '"destination":{'
        '"ip":"$server_addr",'
        '"port":$server_port'
      '},'
      '"nginx":{'
        '"request_time":$request_time,'
        '"upstream_connect_time":$upstream_connect_time,'
        '"upstream_response_time":$upstream_response_time,'
        '"upstream_header_time":$upstream_header_time'
      '}'
    '}'
  '}';

  ...
}

server {
  ...

  access_log /var/log/nginx/radsite-access.log json_datadog;

  ...
}

Which creates output in the DataDog dashboard like:

{
 "http":{
    "status_code":200,
    "status_category":"OK",
    "content_type":"application/json",
    "referrer":"",
    "url":"/some/path?user_id=20k40ffk",
    "url_details":{
       "path":"/some/path",
       "queryString":{
          "user_id":"20k40ffk"
       }
    },
    "method":"GET",
    "request_id":"d4f70c20f1c9cf8263753e601c0f3594",
    "useragent":"CFNetwork/976 Darwin/18.2.0",
    "useragent_details":{
       "os":{
          "family":"iOS",
          "major":"12"
       },
       "browser":{
          "family":"safari",
          "major":"3"
       },
       "device":{
          "family":"iOS-Device",
          "model":"iOS-Device",
          "category":"Mobile",
          "brand":"Apple"
       }
    },
    "x_forwarded_for":"1.1.2.2"
 },
 "network":{
   "bytes_written":928,
   "bytes_read":879,
    "client":{
       "ip":"10.10.10.2",
       "port":30020
    },
    "destination":{
      "port":443,
      "ip":"10.10.10.3"
    },
    "nginx":{
       "request_time":0.025,
       "upstream_connect_time":0.004,
       "upstream_header_time":0.024,
       "upstream_response_time":0.024
    }
 },
 "timestamp":"1571897461960"
}

Occasionally a yum update restores conf.d/default.conf on my CentOS 7 installs, and other times I just need to remove a site from its current server. My Nginx role in Ansible creates and updates server definitions for me, but I wanted the option to wipe out any configs I hadn't specifically defined for a server. It would take care of both my above cases, as well as any other site configs that may have snuck their way into my server, say if I had been testing something and left a config behind.

In the role defaults/main.yml I use a boolean that defaults to no for removing unmanaged sites. I like having to explicitly enable this behavior for each server since it is destructive.

In the first task I run a basic find command to locate all files regardless of extension in the Nginx config dir. I don't want anything but active configs in there. It is idempotent so allowed to run even in --check mode.

The second task required building the right when: filter, which was done with a little guidance from here and here. My Nginx role mentioned above uses a dict with the base name of each config (ie: myapp) as the keys. We pass the keys into the Jinja2 filter that appends .conf to each key, then returns the modified keys as a list in the format: [myapp.conf, othersite.conf, ...]. With that list in hand it is easy to loop over the output of our find command and any filenames found which don't match our key list take a trip to our 51st state: absent. Get it? I'll see myself out.

# setting in role defaults
nginx_remove_unmanaged_sites: no

# Find every file in the conf.d dir
# Allow to run in check mode, mark task as "never changed"
- name: Find existing site configs
  shell: find /etc/nginx/conf.d -type f -printf "%f\n"
  register: contents
  when: nginx_remove_unmanaged_sites
  check_mode: no
  changed_when: no

# remove files found above that aren't in nginx_sites
# append '.conf' to each key in nginx_sites with some regex magic
- name: Remove unmanaged configs
  file:
    path: "/etc/nginx/conf.d/{{ item }}"
    state: absent
  with_items: "{{ contents.stdout_lines }}"
  when: nginx_remove_unmanaged_sites and item not in nginx_sites.keys()|map('regex_replace','^(.*)$','\\1.conf')|list
  notify:
    - reload nginx