Proposal: Public Suffix API

[mckenfra]

This formalizes #231 into a concrete proposal.

[oliverdunk]

Thanks for this! I will reach out to the PSL maintainers I have been in contact with to ask them to take a look. I'll also share this internally to get an overall opinion from Chrome.

[Rob--W]

I'm not too concerned about the performance aspect of this; any IPC to handle the API call is likely more expensive than a look up in the PSL.

The proposal does not describe why unknown suffixes should be supported. Could you elaborate when one would want and when one does not want public suffixes? And offer a recommendation on default behavior that is not dependent on (likely not relevant) performance considerations?

It's probably worth pointing out explicitly that there may be web pages displayed in the browser that are not on a registrable domain, e.g. when a local intranet has custom non-public host names.

[mckenfra]

I will update this section regarding unknown suffixes as you suggest.

On the performance point: I am concerned about performance of this API, because if it turns out to be slower than extensions' own public suffix implementations, then they may not use this API. In particular, Use Cases 1 (Filter Requests by Organization) and 3 (Detect Third-Party Requests) in this proposal are performance-sensitive, because they obtain the registrable domain for every request (not just top-level pages) as the user browses.

[simon-friedberger]

If it's faster for the extension to use an internal list, wouldn't that imply that we should really improve our implementation?

[Rob--W]

Why are Unknown suffixes required in this case? Theoretically better performance does not automatically translate to a functional requirement.

[mckenfra]

Assuming there is no performance benefit from allowing unknown suffixes, I will revisit this requirement.

[Rob--W]

Do we really need all three options (domain, base, private)?

• (input) domain is redundant, because the extension can keep track of the parameters that it had sent to the API.
• Requiring base and private may require two lookups in the PSL, even if the extension does not need it. An extension interested in both can call the API twice with excludePrivateSuffixes set to true and false.

Given this, what do you think of reducing the number of options to just one registrableDomain?

[mckenfra]

Calling the API twice with excludePrivateSuffixes set to true and false has the following potential issues:

Duplication of work

• Must parse/canonicalize the same input domain parameter on each API call.
• The PSL lookup involves removing each label in turn from the input domain and testing the remaining suffix until a match is found. Therefore on the second API call, the same unmatching candidate suffixes that have already been tested in the first API call will again be retested. The difference will be that the algorithm keeps going further in terms of removing labels with the excludePrivateSuffixes=true call.

Duplication of returned arrays

• If the registrable domains array obtained with excludePrivateSuffixes=false happens to only contain ICANN domains (because no matching private suffixes exist), then the second API call with excludePrivateSuffixes=true will return exactly the same registrable domains array again.

Note that my proposal does not require "two lookups". One possible implementation would simply continue removing labels after finding a private suffix until the ICANN suffix is found, i.e. not a full lookup, but a continuation of the current lookup. Even better, a more optimal implementation would have the corresponding ICANN suffix pre-calculated and stored with every private suffix, such that no further work would be needed to determine the ICANN suffix upon matching a private suffix.

I am happy to remove domain in the result, because I agree this can be inferred from the array position.

[Rob--W]

What is the source for the required assumption of "single-label" suffix?

This proposal does not include an example where known vs unknown matters, but in Firefox specifically, there is at least one example where it matters (https://bugzilla.mozilla.org/show_bug.cgi?id=1621168): In determining whether to issue a search query or whether to try a navigation, a PSL lookup is made:

• If valid, attempt to navigate.
• If invalid, use search engine.

Unknown entries in the PSL should also trigger a search query, but unconditionally making it return a single label would rule out that use.

[mckenfra]

I am happy to remove this assumption, and instead add an option to the API allowing users to opt-in to this behaviour of assuming a single-label suffix. I was guided in my analysis by the following:

• Google Chrome's Public Suffix implementation makes this assumption, I believe. See GetDomainAndRegistry: "If no matching rule is found in the effective-TLD data (or in the default data, if the resource failed to load), the last subcomponent of the host is assumed to be the registry."
• tltds also makes this assumption in its implementation. (tltds is a javascript library for obtaining Public Suffixes that is used by popular extensions such as bitwarden password manager and violentmonkey.) However, there is also an open tltds issue caused by this assumption.
• Use Cases 1 (Filter Requests by Organization) and 3 (Detect Third-Party Requests) in this proposal are performance-sensitive, because they obtain the registrable domain for every request (not just top-level pages) as the user browses. I wanted to reduce the chance of making users' browsers feel permanently slower after installing these sorts of extensions. This assumption of a single-label suffix by default may allow a faster implementation due to avoiding an explicit PSL lookup for all single-label TLDs.
• Use Case 2 (Group Domains in UI) in this proposal may benefit from this assumption, because without it all unknown-suffixed domains would be grouped together.

[remusao]

Hey, I'm the maintainer of tldts and wanted to clarify that the issue linked is about tldts-experimental, a faster but less accurate implementation of tldts. The main tldts package (which most projects are using) is fully accurate, and behaves in a slightly different way by returning a single label in case no rule was found matching the last label (or anything more specific), but will also return two flags: isIcann and isPrivate, which allow to know where the matching rule came from (i.e. the ICANN or PRIVATE section of the public suffix list). If both of these flags are false, then we can assume that the fallback behavior of using the last label as public suffix was triggered. For example:

$ npx tldts https://example.thissuffixdoesnotexist
{
  "domain": "example.thissuffixdoesnotexist",
  "domainWithoutSuffix": "example",
  "hostname": "example.thissuffixdoesnotexist",
  "isIcann": false,
  "isIp": false,
  "isPrivate": false,
  "publicSuffix": "thissuffixdoesnotexist",
  "subdomain": ""
}

I lack context on this spec but this might allow to combine the single-label suffix fallback behavior with the use-cases outlined by @Rob--W above.

[simon-friedberger]

The main use-cases for the PSL are isValidUrl(...) which does not want this behavior (because you want to know if the TLD is known) and isSameSite(...) for which it almost doesn't matter.

It only helps if you have made up TLDs. printer.homenet and backup.homenet would both have an empty eTLD and eTLD+1 would be "homenet". If you assume the last label must be a TLD then you end up with the correct behavior.

However, since most browsers currently do not update their PSL, the "made up TLD" situation can also arise when a new TLD gets introduced.

I think the correct solution here is to make the API isValidURL(...) check the last label and isSameSite(...) to assume that it is a valid TLD. But maybe there are use-cases that I'm not seeing which need a third option.

[Rob--W]

Why should the API convert the domain to Unicode by default? When the input is the host name from a URL, it would be reasonable to expect a valid hostname. I'd argue that punycode is the more sane default.

Extensions can bundle the custom library/logic to convert punycode to unicode if desired, because the algorithm is well known and fixed.

[mckenfra]

I can change this to returning punycode by default, with an API option to convert to unicode. Unicode registrable domains are required by Use Case 2 (Group Domains in UI) in this proposal.

[simon-friedberger]

Maybe stress that in almost all cases you want to use the full PSL. If you're trying to attribute abuse at some point you want to go from "a.foo.com" and "b.foo.com" and "c.foo.com" are all distributing malware to "foo.com" is distributing malware. That's the only reason I can think of, though and maybe that should instead be a generic "attribute to all higher labels".

[simon-friedberger]

It's called a public suffix so calling it a private suffix here might be a bit confusing. Private section suffix?

[simon-friedberger]

There can be multiple suffixes per domain but the distinction between private section and ICANN section is a bit arbitrary. There could be multiple entities acting as "registrars" in a domain.

[simon-friedberger]

nit: delete "it"

[simon-friedberger]

I think this might be a made up requirement. (That wiki page really needs some work.) The algorithm is pretty trivial and just needs to compare labels for equality so any encoding should be fine.

[simon-friedberger]

This would probably be a mistake. There are unresolved, lengthy, ancient discussions on this elsewhere but basically, at the moment, public suffixes sometimes have websites.

[simon-friedberger]

The PSL exists because filtering by organization requires the information on the PSL so only using the ICANN section seems just wrong here.

[simon-friedberger]

Another reason why I dislike this is, that the distinction between an ICANN/IANA TLD and an eTLD is - in practice - poorly defined.
The "official" ICANN/IANA TLD is only the last label. There is an IANA list and another list for gTLDs.

However, the PSL is trying to gather information from the official registrars and include that. To maintain the PSL we try to get in touch with the registrars to get them to update their "official" subdomains or to ask questions but we often fail.

Look at the.au section of the list, they cannot agree what their TLDs are but they are in the ICANN section. .bd even has a wildcard *.bd entry in the ICANN section. .co.uk is in the ICANN section but the actual IANA TLD is still only .uk.

[simon-friedberger]

Afaiu this would only make a difference when given a single label. But a single label is either not a valid URL or it is a TLD. If you're trying to look up isValidURL(...) you don't want the optimization because it breaks the use-case. If you're looking up isSameSite(...) then the optimization for a single label is irrelevant because you have to compare them anyway.

[simon-friedberger]

Are callers gaining anything from calling this with an array instead of calling the other function multiple times?

[simon-friedberger]

I've dug into PSL use-cases a couple of time now. I think it might be better to make the API
isValidURL(...) and isSameSite(...). That should be enough for use-cases (1.) and (3.) above. For use case (2.) you're restricted to comparison based sorting algorithms but that's probably okay in practice.

[simon-friedberger]

@remusao, if you have feedback on this I would love to hear it!

[simon-friedberger]

As @Rob--W just pointed out in chat, there are some user-interface use-cases which would still need the getRegistrableDomain function. For example, highlighting in the URL bar or giving a name to a group of same-site URLs.

I still think it would be good to have an explicit API for the two cases mentioned above.

[simon-friedberger]

I support doing it this way but we will have to talk about it because that is not what browsers are currently doing.

That situation has been unresolved for a long time and people have been working around it or ignoring it. I suggest we simply defer to what the browsers are already doing here because that is what we want.

[simon-friedberger]

This is important to keep in mind.isSameSite("100.200.30.2", "100.200.31.2") should probably return false so even if the last label is assumed to be a TLD (and thus not checked) there should be an exception that checks for IPv4 addresses.

[simon-friedberger]

This seems fine but it should explain why the browser is not more fingerprintable, for example, by arguing that the browser version is already available.

[simon-friedberger]

As stated above, it does expose which version of the PSL is used.

[simon-friedberger]

This is a bit at odds with the desire to retrieve version information.

[oliverdunk]

@simon-friedberger let me know that the PSL has been updated with a VERSION metadata item like this: publicsuffix/list#1808 (comment)

Should we mention that here and agree on using that as the version we return?

[Rob--W]

I'd be fine with that.

I am also fine with leaving the version method off altogether if it were to be a blocker to approving the proposal.

After all, there are many existing APIs that are dependent on the result of the public suffix list without exposing versioning information.

[oliverdunk]

I definitely agree that this shouldn't be a blocker. However, I hope it will be non-controversial. I expect most (likely all) browser vendors don't have easy access to this information today but it should be fairly trivial to get access to.