Issues with Local DNS

[ccravens]

I have a ziti controller hosted on a local network with a domain served by a local DNS server. In my example below zt.example.com is a ziti controller on a local network with no availability on the internet and local network DNS as the DNS authority for example.com served at 192.168.0.53

The problem is when I try and import identities from this local ziti controller and local DNS server into my Mac client, they enroll but never fully complete and turn green. Screenshot below of it staying red even though it enrolled:

In the appex.log at first everything looks fine:

(11169)[2026-01-29T06:40:17.264Z]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://zt.example.com:443] controller initialized
(11169)[2026-01-29T06:40:17.264Z]   DEBUG ziti-sdk:ziti_ctrl.c:648 ziti_ctrl_init() ctrl[https://zt.example.com:443] ziti controller client initialized
(11169)[2026-01-29T06:40:17.264Z] VERBOSE ziti-sdk:ziti_ctrl.c:145 start_request() ctrl[https://zt.example.com:443] starting GET[/version]
(11169)[2026-01-29T06:40:17.264Z]    INFO ziti-sdk:ziti.c:606 ztx_init_controller() ztx[3] Loading ziti context with controller[https://zt.example.com:443]
(11169)[2026-01-29T06:40:17.264Z]   DEBUG ziti-sdk:ziti.c:537 ziti_start_internal() ztx[3] using metrics interval: 0
(11169)[2026-01-29T06:40:17.264Z]   DEBUG ziti-sdk:ziti.c:226 ziti_set_unauthenticated() ztx[3] setting auth_state[0] to 0
(11169)[2026-01-29T06:40:17.264Z]   DEBUG ziti-sdk:ziti_ctrl.c:380 ziti_ctrl_clear_api_session() ctrl[https://zt.example.com:443] clearing api session token for ziti_controller
(11169)[2026-01-29T06:40:17.264Z]   DEBUG ziti-sdk:ziti_ctrl.c:1106 ctrl_paging_req() ctrl[https://zt.example.com:443] starting paging request GET[/external-jwt-signers]
(11169)[2026-01-29T06:40:17.264Z] VERBOSE ziti-sdk:ziti_ctrl.c:1111 ctrl_paging_req() ctrl[https://zt.example.com:443] requesting /external-jwt-signers?limit=25&offset=0
(11169)[2026-01-29T06:40:17.264Z] VERBOSE ziti-sdk:ziti_ctrl.c:145 start_request() ctrl[https://zt.example.com:443] starting GET[/external-jwt-signers?limit=25&offset=0]
(11169)[2026-01-29T06:40:17.272Z] VERBOSE tlsuv:http.c:411 client not connected, starting connect sequence
(11169)[2026-01-29T06:40:17.272Z] VERBOSE tlsuv:http.c:411 client not connected, starting connect sequence
(11169)[2026-01-29T06:40:17.272Z] VERBOSE tlsuv:http.c:411 client not connected, starting connect sequence
(11169)[2026-01-29T06:40:17.277Z]   TRACE tlsuv:connector.c:223 fd[18] connecting to X.X.X.X
(11169)[2026-01-29T06:40:17.277Z]   TRACE tlsuv:connector.c:223 fd[16] connecting to X.X.X.X
(11169)[2026-01-29T06:40:17.277Z]   TRACE tlsuv:connector.c:223 fd[20] connecting to X.X.X.X
(11169)[2026-01-29T06:40:17.279Z]   TRACE tlsuv:connector.c:223 fd[22] connecting to X.X.X.X
(11169)[2026-01-29T06:40:17.279Z]   TRACE tlsuv:connector.c:223 fd[23] connecting to X.X.X.X
(11169)[2026-01-29T06:40:17.279Z]   TRACE tlsuv:connector.c:223 fd[21] connecting to X.X.X.X

But then I start seeing this:

11169)[2026-01-29T06:40:17.316Z] VERBOSE tlsuv:http.c:261 src connected status = -3008
(11169)[2026-01-29T06:40:17.316Z]   DEBUG tlsuv:http.c:282 failed to connect: -3008(unknown node or service)
(11169)[2026-01-29T06:40:17.316Z]    WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://zt.example.com:443] request failed: -3008(unknown node or service)
(11169)[2026-01-29T06:40:17.316Z]    WARN ziti-sdk:ziti_ctrl.c:336 internal_version_cb() ctrl[https://zt.example.com:443] CONTROLLER_UNAVAILABLE(unknown node or service)
(11169)[2026-01-29T06:40:17.316Z]    WARN ziti-sdk:ziti.c:2042 version_pre_auth_cb() ztx[3] failed to get controller version: CONTROLLER_UNAVAILABLE/unknown node or service
(11169)[2026-01-29T06:40:17.316Z]   DEBUG ziti-sdk:ziti.c:1704 do_ztx_set_deadline() ztx[3] expire_cb[ziti.c:pre_auth_retry] timeout[5000]
(11169)[2026-01-29T06:40:17.316Z]    WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://zt.example.com:443] request failed: -3008(unknown node or service)
(11169)[2026-01-29T06:40:17.316Z]    INFO ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[https://zt.example.com:443] attempting to switch endpoint

I'm wondering if it has something to do with matchDomains:

ziti-tunnel-apple/Network/PacketTunnelProvider.swift

Line 171 in 3bb60db

func updateTunnelNetworkSettings(_ completionHandler: @escaping (Error?) -> Void) {

I'm thinking that at first it uses the local DNS 192.168.0.53 for resolving the zt.example.com controller address, but once it enrolls it starts using the ziti DNS at 100.64.0.2 and because the DNS authority is local (not publicly available) it's not able to resolve it and errors in the middle of the process.

[scareything]

Hi,

The macOS resolver uses multiple DNS servers to resolve hostnames. The DNS server that is selected for resolving a given hostname is determined by the domain name of the host. You can see the DNS servers that will be selected for each domain with scutil --dns:

% scutil --dns
DNS configuration

resolver #1
  search domain[0] : localdomain
  nameserver[0] : 2605:9480:158:bcd0::1
  nameserver[1] : 10.0.1.1
  if_index : 17 (en19)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #3
  domain   : mattermost.ziti.com
  nameserver[0] : 100.64.0.2
  if_index : 44 (utun8)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102211

...

I mention this to point out that macOS can (and should) use your 192.168.0.53 and the tunneler's 100.64.0.2 DNS servers simultaneously, macOS will use the DNS server from the resolver that has the closest matching domain. So if macOS is using the tunneler's DNS server to resolve a .example.com hostname, then I'm guessing that one of the the OepnZiti services available to your identity has an intercept address of "*.example.com" (or possibly just "example.com" ). The tunneler tells macOS which domains its DNS server knows how to answer for by passing them in the matchDomains when creating the network interface that intercepts packets for OpenZiti services.

Do you have services that try to intercept example.com hostnames?