Automation of PRIVATE section removals

[dnsguru]

This project has evolved a fairly strong ingress process for adding entries.

Once folks get their entry/ies added, it is typically a set and forget activity until they want to amend or modify them.

Few if any submitters come back to remove their entries once set or functioning.

We have put in place a requirement on new submissions to the PRIVATE section that 2+ years of service before expiry are required, in anticipation of some form of culling automation using the expiry date being a data point used to determine the domain is still 'on the air'. Another data point might be the ongoing presence of a _psl txt record where possible.

Without being prescriptive of the sensors and triggers for removals, there seems a need to have some automated deletion process that can scan and use some agreed logic for parsing and removing entries.

Long version:
The PSL gets adds and updates, but typically few or no deletions. One exception is the ICANN contracting json update automation for adding and removing TLDs in the ICANN section of the file, but an area that is growing in size which would benefit from attention is the PRIVATE section.

The file size has grown, and continues to grow while the PSL is leveraged as a fast-fail on inclusion or other derivative use cases that seem to require a PSL entry to get some special handling or status.

The volunteers suspect that there are numerous domains that are in the list which have not been renewed or have expired and been re-registered/drop-caught or have been acquired in the secondary market prior to their expiry by third parties.

This project is staffed by volunteers, all of whom are not looking to have more volunteer time extracted to deal with more than the current process of adds/modifies, which has been greatly aided by automation with the test scripts and the root zone deltas.

Would like to discuss.

Is there consensus that we need deletion automation within the PRIVATE section, and:

• frequency
• sensors/trigger conditions
• overrides
• phases

[dnsguru]

So far, there are a number of ccTLDs that do not present the expiration date in automated lookups.

Whatever automation might have to look to those domains differently for validation of existence and frequency of flagging/purges

[dnsguru]

I am going to run a manual sweep on the private section to determine expired domains for removal using RDAP response expiry dates on gTLDs, and will report the results in aggregate here.

[fsodic]

@dnsguru Is this a discussion? if so, there may be better solutions for forming a healthier "private section". Like looking for tools that can "dig" _psl automatically for domains in this private section.

[dnsguru]

Fajar it is absolutely a discussion about how to get this done - and appreciate your input! Lookups for the presence of a _psl txt record that persists could be a strong way to do this review faster... Because the existence or non-existence of the domain in the registry does not take into consideration that it is still registered by the same submitting party or even registrant, or is even connected in DNS in the same manner or intent that was part of the PR or rationale for inclusion into the PSL. Doing this in DNS would work well, actually. Most folks seem to move on to whatever else they were doing after requesting inclusion, and the PSL entry becomes 'set and forget'. Because of the validation step where a _psl txt record gets added in order to qualify an entry, one could reckon that just leaving that in place is simpler, as it takes another step to remove the entry from DNS. Then, we leave the entry while the _psl txt record remains. Interesting to consider.

…

[fsodic]

It might be a problem when the domain does not have a psl for reasons that a dns error coincides with it and dig might detect the absence of the dns as a sign the domain is inactive.
So a better solution is to add 2 or 3 files that hold dns data from an existing domain separated into 3 different levels.
Orange or Yellow: for a light warning flag
Red: for intermediate deletion warnings
Black: for the last warning
and the last crawl is permanent deletion.

and an additional solution is with the psl which is scheduled for 1 week so that there is a 1 month chance to fix their dns psl.

Another part is when the domain falls into each category there may be a reminder email to the sender of the psl that they include in the registration of the person in charge.

[fsodic]

It might be a problem when the domain does not have a psl for reasons that a dns error coincides with it and dig might detect the absence of the dns as a sign the domain is inactive.
So a better solution is to add 2 or 3 files that hold dns data from an existing domain separated into 3 different levels.
Orange or Yellow: for a light warning flag
Red: for intermediate deletion warnings
Black: for the last warning
and the last crawl is permanent deletion.

and an additional solution is with the psl which is scheduled for 1 week so that there is a 1 month chance to fix their dns psl.

Another part is when the domain falls into each category there may be a reminder email to the sender of the psl that they include in the registration of the person in charge.

And the auto removal <1 year isn't need anymore when the PSL still active.
Every approve merger, make sure the submitter must keep the psl in their dns domain.

i'm sure our PSL be better with this system model.

[frknakk]

We have put in place a requirement on new submissions to the PRIVATE section that 2+ years of service before expiry are required

I ran into a few problems with this requirement that I'd like to mention here:

• There are TLD's where the expiry date is not publicly retrievable (e.g. german .de domains).
• Providers that are paid by invoice often don't have an option to specify a domain expiration date. The domain is simply extended every year by 1 year until the contract gets cancelled by the customer.

Of course you can avoid both problems by switching to other TLD's and other providers. I just wanted to mention that it could restrict many people :)

[dnsguru]

We have seen a number of TLDs that do not provide the expiration date in the public whois/rdap. In addition to .de, I have seen this to be the case with .je/.gg, .ru, .nl, and .pl as well, and I am certain there are others we will find. The invoice payment scenario, as well as others that we have not heard of, whereby the 2 year + term may not be possible will certainly exist. We can make a catch-all, but it doesn't allow for automation to really work... The solution seems to be that the submitting party just acknowledges that their entries can be subject to removal if not maintained through committing to keep the domain name(s) submitted renewed in their PR.

…

On Wed, Dec 16, 2020 at 6:21 PM Furkan A. ***@***.***> wrote: We have put in place a requirement on new submissions to the PRIVATE section that 2+ years of service before expiry are required I ran into a few problems with this requirement that I'd like to mention here: - There are TLD's where the expiry date is not publicly retrievable (e.g. german .de domains). - Providers that are paid by invoice often don't have an option to specify a domain expiration date. The domain is simply extended every year by 1 year until the contract gets cancelled by the customer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1119 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQTJK4SVPNCHP4X5WGNBTSVFTL5ANCNFSM4SJM3HNA> .

[pereceh]

I think the requirements on the wiki are pretty good.

just use the dns _psl records as a benchmark for mass deletion... then give time if the user's DNS server changes to re-create those _psl DNS records (using their old values)...

[wdhdev]

@simon-friedberger I think we can close this, most of this stuff will have to be done manually (it will likely be quite difficult to automate, which we don't have the resources for) and most clean ups are already being done by @groundcat and I.

[simon-friedberger]

I generally agree. The reason I left this open is that I have a concrete idea of something I would like us to do:

Take the psltool output - which can already check the entire list - and turn it into a diff/changelog.

Something that shows us: "last week these domains dropped their _psl entry". I think that would be useful information to have for the manual work and it is transient information that is not captured anywhere. And it is not particularly hard to do.

1. Make psltool output something that can easily be diffed and parsed.
2. Make a github action that runs once a day and stores the output somewhere (probably in the website repository)
3. Make a github action that runs once a day and takes the last two outputs and turns them into something like

Domain	_psl present	_psl entry changes
example.com	Yes	Removed 2024-11-27, Added 2024-11-28
...

Because if we ever want to implement a policy like "if the _psl record is missing for a month we remove your entry" we will need this.

[dnsguru]

A far better approach would be to make entries be in a registry with expiration dates - requiring renewal attention like domains

[simon-friedberger]

Can you elaborate on why that would be so much better?

I'm not strongly against it. It's a little bit of work but we have the date at which an entry was added and can send out an email warning people that their entry is about to expire and force them to click on a link to keep it. The requred back-end would be fairly minimal. It is certainly work that somebody would have to do but maybe not much, much more work than my proposal.

So if you have good reasons, I am all ears.

[dnsguru]

Can you elaborate on why that would be so much better?

I fall back to the basic premise of the two-problems and WHY the PSL is the least awful solution available to them. This seems to have caused the PSL to become the go-to resource for so many as a means to solve those problemts.

PROBLEM 1: Domain Administrators need a method to articulate and define the manner in which they would prefer the internet interact with a given domain name.

PROBLEM 2: Application Developers and Systems seek a more sophisticated means to interact with domian names than is available with DNS.

Making a change like this to the PSL, to introduce a lifecycle, is a preparatory phaze that would help align PSL request/maintainence with how domain name lifecycles work elsewhere.

Domain managers are accustomed to annual renewals, so this just follows that familiar practice.

IF we do this, it should be considered holistically in a larger plan to align with how domain names are managed elsewhere and introduce far more sophisticated and intentional intake that allows for better articulation of The benefits would be numerous.

• Changing this would pivot the "set-and-forget" (SaF) mindset to a more mindful one with orgs / individuals requesting entries. I think the SaF is the heaviest contributor to "debris" in the PSL.
• Having a contact per entry could be more robust than potentially anonymous github Being able to collect better details about the requesting org in order to contact them for follow-up for things like reviews or notice their _PSL DNS entry isn't present etc would be a benefit. (We might need to relocate outside of EU to not have to deal with the fierce overhead of GDPR on this)

The vision for this is that it should be set up as a registry / registrar type architecture with an SRS/EPP model used so that it is something that a domain manager / owner administrator could request through their registrar vs some minimal thing, though. EPP Registries have a 'pending create' status that creates (But does not guarantee creation) that we as volunteers could then turn off to allow / reject entries using. Upon approval, the appropriate PR would be generated and directions to the requesting party on the _PSL TXT record would be furnished and verified.

We would / could have questions per domain name that are very specific and intentional about what the desired behaviors of each domain name are.

(Non-Exhaustive) Examples of specific definitions that could be made, per domain:

• allow/not wildcard ssl ;
• allow/not cookies below ;
• will have 3rd parties registering subdomains (or not)
  • provide abuse contact / lookup for subordinate 'whois' details
• many other questions

Those and other questions would help understand the request so that very intentional lists could be generated.

• Adding a nominal addition fee / renewal cost
  • Could aid in self-funding for resourcing the change in intake for the project
  • Promote self-resistence to 'trial ballon' stuff that spins the wheels of volunteer resources and contributes to SaF debris

[simon-friedberger]

Okay, a couple of obvious points up front:

1. No, we are not charging for being on the list. (cf Better document the PSL roadmap and needs #671 (comment))
2. No, we are not storing any data which could cause privacy issues.

Regarding adding additional questions, we could do that fairly easily. So, if you want to make a case for some of your examples just open issues! I am not doing that because if, for example, CAs wanted to know if people want to allow wildcard certificates, then they should open that issue.

Okay, now let's finally get to your initial point: Automatic expiry!
We could do that fairly easily as well! There is nothing stopping is from telling people: "Please add a comment 'renew' at most 1 month before expiry to your old Github PR otherwise your entry will be automatically removed." and then writing a few scripts that extract this data. The reason I don't want to do this is that it requires manual renewal, while most domains use auto-renewal.

Aligning the PSL with other DNS operations seems like the correct solution but given that DBOUND has failed multiple times and that RWS just ended up with a list similar to this one, I don't see a way forward here.

Except manually trying to extract the information we need from DNS which is basically what we are doing and what my proposal was. In other words, expiry happens when the _psl entry is missing for a prolonged period of time.

[wdhdev]

I think we should just say if a _psl TXT record is missing for around a month or so, we should remove the entry. For existing entries that do not have _psl TXT records (whether they were added before they were implemented, or they have been since removed) we could either:

1. Give them an exception to the rule
2. Give then a period of time (such as 6 months) to add a _psl TXT record. (If they were added before we were on GitHub, maybe include a link to the bug?)

[simon-friedberger]

I think we are already covering that? See #2023

[wdhdev]

Oh, I forgot about that. My main point was regarding older entries, but for newer entries we could easily automate removals. I can create a GitHub action for it if you'd like.

[simon-friedberger]

I don't want to "just" remove them. I think we definitely should send a warning email first. But yes, please create such an action! I will check with our SRE folks on how we can send proper email.

[wdhdev]

I'm not familiar with Go, so I will be using JavaScript, which I hope is fine. We should be able to sort out emails some how.

[mozfreddyb]

(As noted elsewhere, we should not have multiple software stacks for the psl. If you don't know Go, that's fine. We can find someone or you can learn it :P But let's not add more complexity if avoidable.)

[wdhdev]

I do recall that. @danderson could you help out here maybe?

[simon-friedberger]

I would prefer adding this to the Go tool as well.

The complexity here is imho:

1. We need to send emails.
2. We need to store state so the action knows that the _psl entry has actually been missing for a few weeks instead of just looking at the last run.
3. Handling rate-limiting and retrying DNS requests.

Those are all things that we probably want to be able to do with the Go code anyway.

[wdhdev]

2. We need to store state so the action knows that the _psl entry has actually been missing for a few weeks instead of just looking at the last run.

We could potentially open a PR using GitHub actions removing all entries with missing _psl TXT records, and then once a month has passed since it was opened, check if the entries being removed are still missing _psl records, update the PR accordingly, and then we can merge it.

[simon-friedberger]

Technically, yes, you could store data in Github PRs. But it will leave us with always having 31 pull requests open and needing to parse the same list of domains out of them.

We could store the data in a separate repository but putting it in a GCP bucket sounds simplest to me. Other ideas?

[wdhdev]

An S3 bucket sounds over complicated, why not just store them in a TXT file or something in the repo?

[simon-friedberger]

I'm fine with it, if you find that easier.

Should we use a different repository to avoid polluting the history on this one with automatic changes from that action? 🤔 Probably, not worth it, I guess.

[wdhdev]

We could, there is an action that allows you to do that super easily.