Docker “Server Misbehaving” – Root Cause Analysis in 3 Acts.

I faced a curious problem where my Grafana instance suddenly would be unable to connect to the database after working fine for days. The logs indicate the Grafana container attempting to connect on port 53, when MariaDB runs on 3306. Of course, this shows it may be DNS-related.

Act 1: Grafana Logs Error Message

The logs are included below:

grafana-grafana-1  | 2023-09-23T11:50:06.402836740Z [xorm] [info]  2023/09/23 11:50:06.402682 PING DATABASE mysql
grafana-grafana-1  | 2023-09-23T11:50:06.404357301Z logger=tsdb.mysql t=2023-09-23T11:50:06.404252888Z level=error msg="Query error" err="dial tcp: lookup matomo_db_1 on 127.0.0.11:53: server misbehaving"

Affiliate Content Start

CHIFENCHY Portable Bluetooth Speaker with Lights,Powerful Crystal Clear Sound,IPX5 Waterproof,All Day Playtime,AUX&TF-Card Input,Bluetooth 5.3,TWS Paring,Small Wireless Speaker for Outdoor,Gift Ideas

(257)

$25.99 (as of July 25, 2024 14:24 GMT +08:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Antool Wired Earbuds,【Ultra clear calls & no noise】 Headphones Earbuds Wired 3.5mm, in-Ear Earphones with Mic, Compatible with iPhone/iPad/Android/Laptop/PC 【2 Pack】

(4841)

$19.99 (as of July 25, 2024 14:24 GMT +08:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

2 Packs iPhone Headphones Earbuds with Lightning Connector(Built-in Mic & Volume Control)[MFi Certified] Headphones support for iPhone 14/13/12/SE/11/XR/XS/X/7/7Plus/8/8Plus Support All iOS System

(9253)

$12.99 (as of July 25, 2024 14:24 GMT +08:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Affiliate Content End

The Google search

After googling “docker ip lookup failing” and “docker 127.0.0.11:53: server misbehaving” I came across the linked article below for an unrelated issue, however, the commenter suggested using docker network inspect and docker container inspect .

Docker DNS server 127.0.0.11 problem – General Discussions – Docker Community Forums

Is there some way to inspect the Docker DNS server? It’s not behaving as expected. Two different containers running on the same server. Same network (Macvlan), with neighboring public IP addresses. Same resolv.conf: nameserver 127.0.0.11 options ndots:0 I can ping 127.0.0.11 from both containe…

Act 2: Docker Network Inspect Command

The plot thickens…. by taking a closer look at the docker network and containers, I was able to validate that both containers (MariaDB and Grafana) are indeed still on the same docker network.

[
    {
        "Name": "local",
        "Id": "9688511103c6d49d0184f023da3c294d8a5c871305997d5a42cc862a7647a1bc",
        "Created": "2021-09-10T15:24:49.835878943Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "31de96114eec6c5486ece14f7fa29696ab56036000b7e7fe44012a518adedc8f": {
                "Name": "grafana-grafana-1",
                "EndpointID": "6c15ba93aa83c9dfc581a64aabeb6120b26c1f0dcaad643929341bc6b675c040",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "e07e243a014850a5d5d1ebbb157f39552d39d6ac66990a2af113912cc32cebf1": {
                "Name": "matomo-db",
                "EndpointID": "4f4bffd9054cec3fce575b05c333e4d36f3bb9fb920552db59908519b748b2e4",
                "MacAddress": "02:42:ac:12:00:06",
                "IPv4Address": "172.18.0.6/16",
                "IPv6Address": ""
            }
]
...

I looked at the exact hostname and the dash caught my eye because for years my Docker containers used underscores in their names. I double-checked my data source settings in Grafana and sure enough, the hostname uses an underscore. I thought this couldn’t be right why would the host change!??

As it turns out, the Maria DB container is now using dashes matomo-db rather than underscores matomo_db.

Then I realised that the Docker Compose upgrade I performed yesterday from 1.29.2 to 2.22.0 must have resulted in the difference in container names. Seems like an odd decision to cause a breaking change for the sake of using dashes in container names.

Kitchen Multi-Timer Pro

Now you’re cooking

Multi Timer Pro is your ultimate meal prep companion, keeping track of multiple cooking times and making adjustments on the fly. Give it a try today and become a better home cook!