support X-Robots-Tag as a typed http header XRobotsTag

[GlenDC]

It should support the rules as documented in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Robots-Tag

Additionally these rules can also be added:

X-Robots-Tag: noai, noimageai, SPC

You can have one or multiple active, some rules are just a value others are a name: value pair.

By default it can just comma-separate the values, but feel free to make it opt-in to have 1 rule per header (and thus mutliple headers.

[hafihaf123]

Hello,

I’ve already completed one issue in this repository ( #358 ), and I really enjoyed the experience. Would it be possible for me to take on this issue as well? My exams are now finished (hopefully), and I have the time to focus on contributing more.

I’m still building my experience with Rust and open-source contributions, and I think this issue would be a great opportunity to grow while supporting the project. If there’s anything specific I should keep in mind or prepare before starting, please let me know.

Looking forward to your response!

[GlenDC]

Nothing in mind, no. All good.
Welcome back! it's all yours :)

[hafihaf123]

Hello!

I have a few questions about implementing this functionality:

1. Where should the code be written? Based on the structure, I was considering rama-http/src/headers/forwarded/, but I’m not entirely sure if that’s the right place.
2. What should the filename be? Would x_robots_tag.rs be appropriate?

Here’s my proposed plan for implementation:

• Add an enum named XRobotsTagRule to represent the rules, with name-value ones including a &str to hold the value. Alternatively, if the value can be represented by specific structs, I could explore that approach, though I’m uncertain about its feasibility.
• Introduce a XRobotsTag struct that contains a Vec<XRobotsTagRule>.
• Implement the rama_http::headers::Header trait for XRobotsTag.

I’d really appreciate your feedback and corrections on this plan. Please let me know if there’s anything I’ve overlooked or if you have suggestions to refine the approach.

Thanks again!

[GlenDC]

Regarding your question, rama-http/src/headers/x_robots_tag.rs can it be done in, with the type(s) exported in rama-http/src/headers/

[GlenDC]

Regarding the implementation details I have following guidelines:

• check sources like https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Robots-Tag out, also linked in the issue description. Your enum approach makes sense, but:
  • no need to store string values as these values are not strings. So the type can be specific to each case. E.g. some might contain an Option (I would parse -1 as None, as to not give a silly 32-bit range of non-sensical negative numbers after parsing), an enum value (none/standard/large)
  • note also that rules can also be bound to a specific robot name (user-agent)

Where possible I would keep implementation details private. E.g. (but these are just possibilities):

• have easy to use constructors and builder pattern for setting data, this way you don't need to expose internal representation at all. E.g. .for_bot(HeaderValue), .no_snippet(), .max_preview_image(PreviewKind), ... stuff like that

At the same time, it might make sense to come up with a sensible API and typeset and do expose these. That would then look a lot more like the Via header, where you would allow the XRobotsTag header to be comestructed FromIterator where the elements turn into a rule set. You would just need to think how that would come into play with the robot-name possibility in mind.

For encoding/decoding these CSV values you can also get inspired by the crate::headers::util::csv::from_comma_delimited code and its users. That said you might need slightly more complex logic given you have these optional bot-name qualifiers at any position.

I would suggest to try to come up with a PR that shows your approach, even if it's still full of todo!() code. It helps in cases like this to first try to come up with an API you are happy about and we are aligned about. As filling in the details is usually pretty trivial after that.