1 ad-guard multiple external dns with multiple domains

[strelok899]

I have in my lab 2 clusters
1 ad-guard server
now when i setup cluster 1 with ad-guard to publish to cluster1.domain.com all working well.
the problem is when i setup the second cluster to the same ad-guard server with cluster2.domain.com
then both clusters turn down to infinite loop of trying to overwrite each other's settings .

is there any way to make it work?

[muhlba91]

this is a documented limitaion: see https://github.com/muhlba91/external-dns-provider-adguard?tab=readme-ov-file#rule-ownership
the problem is that, using comments, ownersip cannot be implemented, and, in fact, filtering by "instance" is not possible within the provider.

[brianramseyau]

@muhlba91 seeing that external-dns writes TXT ownership records by default (though configurable), why are these not taken into account instead of needing comments?
I run multiple clusters in production environments writing to the same cloudflare zones using the same principles...

[muhlba91]

thank you for your comment. i revisited this one and compared it to the cloudflare in-tree provider:

• cloudflare returns all records in cloudflare to external-dns: https://github.com/kubernetes-sigs/external-dns/blob/master/provider/cloudflare/cloudflare.go#L249 - note: it doesn't perform any filtering, just returns all records.
• external-dns implements the TXT registry. this implementation gets all records from the provider (e.g. cloudflare) and annotates them with the TXT records it finds. https://github.com/kubernetes-sigs/external-dns/blob/master/registry/txt.go#L109
• when sending records to change/create/delete over to the provider, external-dns' TXT registry (https://github.com/kubernetes-sigs/external-dns/blob/master/registry/txt.go#L246) takes care of
  • filtering the changes by owner id: https://github.com/kubernetes-sigs/external-dns/blob/master/endpoint/endpoint.go#L325
  • generating TXT records
  • sending these changes to the configured provider: https://github.com/kubernetes-sigs/external-dns/blob/master/registry/txt.go#L301
  • cloudflare provider takes care of sending these with the correct function to cloudflare (https://github.com/kubernetes-sigs/external-dns/blob/master/provider/cloudflare/cloudflare.go#L272)

looking at this provider now, we return all records (that are potentially managable by any external-dns) in https://github.com/muhlba91/external-dns-provider-adguard/blob/main/internal/adguard/provider.go#L142, and we apply all changes sent over in https://github.com/muhlba91/external-dns-provider-adguard/blob/main/internal/adguard/provider.go#L58. this works as follows:

• we collect all possible endpoints that are managable by any external-dns: https://github.com/muhlba91/external-dns-provider-adguard/blob/main/internal/adguard/provider.go#L72
• we are deleting all targets/endpoints that are to be updated or deleted: https://github.com/muhlba91/external-dns-provider-adguard/blob/main/internal/adguard/provider.go#L92
• we add all new or updated targets/endpoints: https://github.com/muhlba91/external-dns-provider-adguard/blob/main/internal/adguard/provider.go#L111
• we serialize all rules again and append non-managed rules: https://github.com/muhlba91/external-dns-provider-adguard/blob/main/internal/adguard/provider.go#L111

looking at this, i actually don't see a reason why this shouldn't work if external-dns is configured correctly. this provider implements the required functions in the same way as, e.g., cloudflare.

what will happen though is that also rules not managed by this external-dns instance are re-rendered which could cause side-effects with the newly introduced feature of adding important if both instances were configured differently.

for the sake of it, let's assume all external-dns adguard providers are configured equally.
what is still different to cloudflare and others is that adguard doesn't allow creating, deleting or updating one single record, or a set of it, but we must send the entire list each time we want to change them.

@strelok899 could you elaborate more on what you mean with then both clusters turn down to infinite loop of trying to overwrite each other's settings?
i can imagine that there's a race condition when multiple external-dns instances are reading and/or pushing changes to adguard at a similar time. the reason being that we always must read all records to perform changes as well as write all records to perform changes, which still takes some time to perform (and probably propagate?).

@brianramseyau @strelok899 do you see an apparent flaw i missed in this analysis?
i don't have multiple external-dns set up at the moment and will be travelling the next days, but are you be able to try it again with setting different reconciliation times for the external-dns instances? for example, what comes in my mind are 7 minutes and 9 minutes which should avoid most of the times similar operations happening within the same minute or so, at least in the first few runs. what would be interesting is if rules are actually deleted/modified across instances without taking into account rule order - i presume that the rule order will change with each run where changes have to be applied...

[brianramseyau]

I guess, my main point here which I didn't really correctly state... Is that if this provider is surfacing all the required information for external DNS and you have set unique --txt-owner-id per cluster, they shouldn't be fighting over any records regardless which provider is updating the DNS records...

[muhlba91]

in principle they should not interfere with each otherwhen --txt-owner-id is set, assuming they have a unique set of domain records.

"in principle" because adguard only accepts writing the entire list and not single entries.
this, in turn, will lead to both external-dns (providers) writing the entire list whenever changes occur. in such a case, the rule order will be different depending on the external-dns sending the rules to adguard, which shouldn't really matter.

the problem i rather see are race conditions where both external-dns (providers) try to update the entire list at the same, or similar enough, time - as we cannot just set one rule by itself, the entire list of existing and new rules has to be sent.
imo, this would lead to a longer reconciliation time/loop but that's just a guess atm, and nothing that could be changed in the provider itself.