Advanced Docker Logging – Capturing Drupal’s watchdog messages in Graylog

Premise: Capturing Drupal’s watchdog messages in a container

Logging in Docker is complicated enough as it is, but it gets even more complicated when Drupal logging is involved.

For most Drupal sites, logging with the Dblog module is not sufficient due to scale, and instead is being handled with the Syslog module. The result isn’t pretty, but works flawlessly and integrates with rsyslog, which is available for both Debian and CentOS-based operating systems.

Problem: Where did the application logs go?

After dockerizing a Drupal application using the php-fpm docker image, I wondered: where did my logs go? Tailing the Docker logs (docker logs -f container_name) showed me php-fpm requests to Drupal’s index.php, but nothing else. Let’s fix that.

Setting up rsyslog in your Docker container

The php (or php-fpm) image comes with the processes that are advertised in its name: php-fpm. The process list of a running container will show that nothing else is running. No process manager, no logs, just php-fpm master and child processes.

This is by design, as Docker containers are supposed to have a single purpose. However, nothing prevents us from starting any other processes, e.g. rsyslog:

apt-get update && apt-get install rsyslog -y && service rsyslog start

Unlike in a regular operating system, Docker containers requires any installed process to be started manually due to a modified policy in /usr/sbin/policy-rc.d. See this article for some more details.

After installing and starting the syslog deamon, we can see our Drupal logs flowing into /var/log/syslog. You can tail the syslog from within the container or from outside.

Redirecting syslog messages to Graylog 2

If you happen to have the beautiful Graylog 2 logging system running in your network, I’d recommend redirecting your syslog messages to Graylog. This will provide you with a beautiful UI and better search features. For the icing on the cake, you can create an extractor for Drupal’s watchdog-style messages, which splits up the single log message into individual fields.

Required components for Graylog integration

Getting this integration up and running requires quite a few components, so here’s a brief list:

  • A running Graylog instance: Graylog 2 Docker Image
  • rsyslog running in your container
  • rsyslog configuration that redirects messages to Graylog
  • Open ports in your Graylog instance
  • Active Graylog input for syslog messages over UDP or TCP
  • Active extractor for syslog messages

Setup comments

Setting up Graylog 2 is no trivial task, but I will refer to the Graylog 2 Docker image, which at least makes this steps relatively easily in a local development environment. We already talked about getting rsyslog setup in your application container, in our case running Drupal 7.

Now, for the rsyslog configuration: Here’s a good tutorial for modifying rsyslog.conf with the result of all logged messages being redirected to your Graylog instance: Get Messages In — Graylog 2.3.0 documentation. Instead of directly modifying the rsyslog configuration file as described in this article, I’d recommend adding a separate configuration file called graylog.conf into /etc/rsyslog.d/, which makes maintaining the config file easier as it’s separated from the default configuration.

In my setup, I initially forgot to open up port 514 over UPD, which would look like this in your docker-compose file for Graylog.

  graylog:
    image: graylog2/server:2.3.2-2
	  ...
    ports:
      - "514:514/udp"

The final step is setting up the Graylog input for syslog messages over port 514. If you want to extract the information from Drupal’s watchdog messages, you can additionally setup a custom extractor based on Managing Drupal log using Elasticsearch, Logstash, and Kibana that parses Drupal watchdog messages through a regular expression and extracts their information into individual pieces. Here’s a sample Graylog 2 content pack for this configuration:

"inputs": [
  {
    "id": "59f0aa643209f300018f906f",
    "title": "Syslog UDP",
    "configuration": {
      "expand_structured_data": false,
      "recv_buffer_size": 262144,
      "port": 514,
      "override_source": null,
      "force_rdns": false,
      "allow_override_date": true,
      "bind_address": "0.0.0.0",
      "store_full_message": false
    },
    "static_fields": {
    },
    "type": "org.graylog2.inputs.syslog.udp.SyslogUDPInput",
    "global": true,
    "extractors": [
      {
        "title": "Drupal Watchdog",
        "type": "GROK",
        "cursor_strategy": "COPY",
        "target_field": "",
        "source_field": "message",
        "configuration": {
          "grok_pattern": "%{URI:drupal_base_url}\\|%{NUMBER:drupal_log_unixtimestamp}\\|%{DATA:drupal_log_type}\\|%{IPORHOST:drupal_client_ip}\\|%{URI:drupal_request_uri}\\|%{URI:drupal_referer_uri}?\\|%{NUMBER:drupal_userid}\\|%{DATA:drupal_page_link}?\\|%{GREEDYDATA:drupal_log_msg}",
          "named_captures_only": false
        },
        "converters": [

        ],
        "condition_type": "REGEX",
        "condition_value": "\\|\\d*\\|[a-zA-Z0-9_ ]*\\|[0-9.]*\\|",
        "order": 0
      }
    ]
  }
]

 

Screenshot

After all this setup work you can easily view and search through your Drupal logs using Graylog.

Extracted Graylog Drupal Watchdog message