chore: deprecate pylint in favor of ruff

[mistercrunch]

This could be slightly controversial, but submitted as a DRAFT for discussion. After thinking about it, I think we should just move forward with this. Shouldn't be controversial. File under simplify/accelerate builds/devex/tooling.

The case for pylint deprecation:

• mostly covered by ruff: what matters most in pylint is already re-implemented
• pylint is SLOW, ruff is a million times faster - pre-commit / pylint is one of our longest, most expensive CI step
• rules are currently scattered in different places and can conflict at times
• checked in with the Airflow community, they went all in on ruff, deprecated pylint and are extremely happy with the switch

Very relevant is this link that explains what pylint features are covered or not by ruff:
astral-sh/ruff#970

Essentially we can simplify & speed things up here..

If we agree on the direction, the plan is to follow up and enable all the rules, apply ruff check --fix and ruff check --add-no-qa, which will enable the set of rules commented now, fix the obvious, and add a # noqa to the lines that are not immediately fixable.

[michael-s-molina]

@mistercrunch Can we reserve this for the 5.0 breaking window? We'll gather proposals for 5.0 next week. I'll send an email and you could include this. I know that this will break our internal release so using a major release is safer.

[mistercrunch]

Can we reserve this for the 5.0 breaking window

When is this scheduled? Thinking about it though, switching dev tools shouldn't have any impacts on UX/interfaces. From my understanding semver is mostly interface oriented. I'm favoring moving fast on this while I have momentum on improving devex. Though it may have impact on mergeability of cherries, so may be good to wait for that.

Also noting that when I enabling the pylint rules currently commented in my PR using ruff, it finds hundreds lint issues, meaning turning on ruff should up code quality moving forward (my plan is to enable these rules, run --fix, and --add-noqa).

[mistercrunch]

@sadpandajoe curious on your thoughts about timing, for context the follow up PR would touch the backend with minor linting tweaks, maybe 300 LoC across the backend.

[sadpandajoe]

@sadpandajoe curious on your thoughts about timing, for context the follow up PR would touch the backend with minor linting tweaks, maybe 300 LoC across the backend.

@mistercrunch I believe the breaking change window is opening soon. I don't think it'll affect Preset but sounds like it might have an affect on AirBnb?