Docker “Server Misbehaving” – Root Cause Analysis in 3 Acts.

I faced a curious problem where my Grafana instance suddenly would be unable to connect to the database after working fine for days. The logs indicate the Grafana container attempting to connect on port 53, when MariaDB runs on 3306. Of course, this shows it may be DNS-related.

Act 1: Grafana Logs Error Message

The logs are included below:

grafana-grafana-1  | 2023-09-23T11:50:06.402836740Z [xorm] [info]  2023/09/23 11:50:06.402682 PING DATABASE mysql
grafana-grafana-1  | 2023-09-23T11:50:06.404357301Z logger=tsdb.mysql t=2023-09-23T11:50:06.404252888Z level=error msg="Query error" err="dial tcp: lookup matomo_db_1 on 127.0.0.11:53: server misbehaving"

Affiliate Content Start

ROVE R2-4K DUAL Dash Cam Front and Rear, STARVIS 2 Sensor, FREE 128GB Card Included, 5G WiFi - up to 20MB/s Fastest Download Speed with App, 4K 2160P/FHD Dash Camera for Cars, 3" IPS, 24H Parking Mode

(637)

$89.99 (as of November 21, 2024 02:57 GMT +08:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Wireless Earbuds, Bluetooth 5.3 Headphones Bass Stereo, Ear Buds with Noise Cancelling Mic LED Display, IP7 Waterproof in Ear Earphones, 36H Playtime for Laptop Pad Phones Sports Workout, Black

(12980)

$29.99 (as of November 21, 2024 02:57 GMT +08:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Apple Watch SE (2nd Gen) [GPS 40mm] Smartwatch with Starlight Aluminium Case with Starlight Sport Band S/M. Fitness and Sleep Trackers, Crash Detection, Heart Rate Monitor, Retina Display

(731)

$173.88 (as of November 21, 2024 02:57 GMT +08:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)

Affiliate Content End

The Google search

After googling “docker ip lookup failing” and “docker 127.0.0.11:53: server misbehaving” I came across the linked article below for an unrelated issue, however, the commenter suggested using docker network inspect and docker container inspect .

Docker DNS server 127.0.0.11 problem – General Discussions – Docker Community Forums

Is there some way to inspect the Docker DNS server? It’s not behaving as expected. Two different containers running on the same server. Same network (Macvlan), with neighboring public IP addresses. Same resolv.conf: nameserver 127.0.0.11 options ndots:0 I can ping 127.0.0.11 from both containe…

Act 2: Docker Network Inspect Command

The plot thickens…. by taking a closer look at the docker network and containers, I was able to validate that both containers (MariaDB and Grafana) are indeed still on the same docker network.

[
    {
        "Name": "local",
        "Id": "9688511103c6d49d0184f023da3c294d8a5c871305997d5a42cc862a7647a1bc",
        "Created": "2021-09-10T15:24:49.835878943Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "31de96114eec6c5486ece14f7fa29696ab56036000b7e7fe44012a518adedc8f": {
                "Name": "grafana-grafana-1",
                "EndpointID": "6c15ba93aa83c9dfc581a64aabeb6120b26c1f0dcaad643929341bc6b675c040",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "e07e243a014850a5d5d1ebbb157f39552d39d6ac66990a2af113912cc32cebf1": {
                "Name": "matomo-db",
                "EndpointID": "4f4bffd9054cec3fce575b05c333e4d36f3bb9fb920552db59908519b748b2e4",
                "MacAddress": "02:42:ac:12:00:06",
                "IPv4Address": "172.18.0.6/16",
                "IPv6Address": ""
            }
]
...

I looked at the exact hostname and the dash caught my eye because for years my Docker containers used underscores in their names. I double-checked my data source settings in Grafana and sure enough, the hostname uses an underscore. I thought this couldn’t be right why would the host change!??

As it turns out, the Maria DB container is now using dashes matomo-db rather than underscores matomo_db.

Then I realised that the Docker Compose upgrade I performed yesterday from 1.29.2 to 2.22.0 must have resulted in the difference in container names. Seems like an odd decision to cause a breaking change for the sake of using dashes in container names.

Kitchen Multi-Timer Pro

Now you’re cooking

Multi Timer Pro is your ultimate meal prep companion, keeping track of multiple cooking times and making adjustments on the fly. Give it a try today and become a better home cook!