docs: added script to shorten documentation links Fixes #6195 #6265

vjgaur · 2024-10-28T17:57:15Z

Description

Added a script to automatically shorten documentation links and apply it to improve readability of the Polkadot SDK documentation. The script converts long-form documentation links to a shorter format while maintaining the full references at the bottom of each file.

Fixes #6195

Integration

The script is added to substrate/.maintain/ and can be used as follows:

Preview changes:

python3 substrate/.maintain/link-shortener.py --dry-run

Apply changes:

python3 substrate/.maintain/link-shortener.py

// Before:
[crate::reference_docs::blockchain_state_machines]

// After:
blockchain_state_machines

Review Notes

Detailed Implementation Overview

Approach

The script uses a systematic approach to transform documentation links:

File Processing
- Recursively finds all .rs files in docs/sdk/src
- Checks each file for presence of crate:: to optimize processing
- Only modifies files that contain documentation links
Link Pattern Recognition
PATTERNS = [ r'\[crate::reference_docs::[^]+\]', r'\[crate::polkadot_sdk::[^]`

These patterns target specific documentation sections while avoiding false positives.
3. Link Transformation Process
def replacer(match): # Extract full path: [crate::reference_docs::item] full_path = match.group(0) # Remove brackets: crate::reference_docs::item path_without_brackets = full_path[2:-2] # Get item name: item short_name = path_without_brackets.split("::")[-1] # Create new format return f'[{short_name}]({path_without_brackets})'

Reference Management
Collects all unique references during processing
Sorts references alphabetically
Appends them at the end of the file in a dedicated section
Maintains original paths for documentation linking
Safety Features
Dry Run Mode: --dry-run flag shows changes without applying them
Error Handling: Gracefully handles file reading/writing errors
Progress Reporting: Shows which files are being processed
Statistics: Reports number of files processed and modified
Testing
The script can be tested in multiple ways.
Dry run on all docs:
python3 substrate/.maintain/link-shortener.py --dry-run

Test specific directory:
python3 substrate/.maintain/link-shortener.py --path docs/sdk/src/guides --dry-run

Results from Testing

Total files processed: 57
Files modified: 32
Common patterns found and transformed correctly
All references maintained and properly linked
Documentation functionality preserved while improving readability

Implementation details:

Script identifies documentation links using regex patterns
Processes .rs files in docs/sdk/src by default
Maintains all original references while improving readability
Includes dry-run mode for safe testing
Files processed: 57
Files modified: 22
No TODOs remaining.

Checklist

My PR includes a detailed description as outlined in the "Description" and its two subsections above.
My PR follows the labelling requirements (awaiting maintainer labels)
I have made corresponding changes to the documentation (this PR is documentation related)
I have added tests (script includes --dry-run option for testing)

Polkadot Address 👍

polkadot address: 14anwfyV1RYtHZjirHpfjAGaSQTirpYagGi8JaXCyy8fr3So

kianenigma · 2024-10-28T18:11:06Z

docs/sdk/src/guides/enable_elastic_scaling_mvp.rs

@@ -135,10 +135,14 @@
 //!      ```
 //!     Note: for a parachain which measures time in terms of its own block number, changing block
 //!     time may cause complications, requiring additional changes.  See here more information:
-//!     [`crate::guides::async_backing_guide#timing-by-block-number`].
+//!     [`async_backing_guide#timing-by-block-number`](crate::guides::async_backing_guide#timing-by-block-number).


Suggested change

//! [`async_backing_guide#timing-by-block-number`](crate::guides::async_backing_guide#timing-by-block-number).

//! [`async_backing_guide#timing-by-block-number`].

I think for these final display should be like this, not containing the header.

Thanks for your suggestion.
I have incorporated those changes.
Let me know if thats fine

kianenigma · 2024-10-28T18:11:40Z

docs/sdk/src/guides/mod.rs

@@ -12,7 +12,7 @@
 //!
 //! > By this step, you have already launched a full Polkadot-SDK-based blockchain!
 //!
-//! Once done, feel free to step up into one of our templates: [`crate::polkadot_sdk::templates`].
+//! Once done, feel free to step up into one of our templates: [`templates`](crate::polkadot_sdk::templates).


Suggested change

//! Once done, feel free to step up into one of our templates: [`templates`](crate::polkadot_sdk::templates).

//! Once done, feel free to step up into one of our templates: [`templates`](crate::polkadot_sdk::templates).

Suggested change

//! Once done, feel free to step up into one of our templates: [`templates`](crate::polkadot_sdk::templates).

//! Once done, feel free to step up into one of our templates: [`templates`].

So in all of them, you should now be able to remove the part that is inside the ().

@kianenigma
Thanks for your suggestion.
I have incorporated these changes.
Let me know if thats fine or anything else needs to be changed further

Hi @kianenigma I am trying to get through pr checks for prdoc
I am not sure what should go in crates because this script isn't impacting any crates but the documentation only. Initially I added substrate since it was giving me error later removed it than it asks for crates to be added , Kindly suggest what crates to be added in prdoc
Required { path: "/crates", },

with the label I have added, it should be good.

Thanks @kianenigma

kianenigma

Looks neat overall! I've spotted two small issues in the first few ones, will do a final review once they are also fixed.

Please provide your polkadot address for a tip.

docs/sdk/src/external_resources.rs

kianenigma · 2024-11-05T11:12:08Z

docs/sdk/src/external_resources.rs

-
-// Link References
-// [`templates`]: crate::polkadot_sdk::templates
+//! Link References


Suggested change

//! Link References

And this line doesn't have to be here, because what comes after is not rendered.

Done fixed it.

kianenigma · 2024-11-15T07:52:50Z

docs/sdk/src/external_resources.rs

@@ -12,3 +12,6 @@
 //! - [Polkadot Developers](https://github.com/polkadot-developers/)
 //! - [Polkadot Blockchain Academy](https://github.com/Polkadot-Blockchain-Academy)
 //! - [Polkadot Wiki: Build](https://wiki.polkadot.network/docs/build-guide)
+


Suggested change

kianenigma · 2024-11-15T07:53:27Z

Is the branch up to date? Have you run the script, and pushed the code?

I still see a lot of wrong updates.

kianenigma · 2024-11-15T07:54:01Z

Most importantly, cargo doc is failing. Are you building this yourself locally?

https://github.com/paritytech/polkadot-sdk/actions/runs/11757781965/job/32755463592?pr=6265

vjgaur · 2024-11-16T10:19:27Z

Most importantly, cargo doc is failing. Are you building this yourself locally?

https://github.com/paritytech/polkadot-sdk/actions/runs/11757781965/job/32755463592?pr=6265

Initially, I did not run cargo doc I was mostly running the script and verifying the changes its making and pushing the code to see if all github checks are passing or not cargo clippy was breaking so was working to fix that.
Thanks for pointing I will do cargo doc will fix issues it its failing

vjgaur · 2024-11-16T10:20:48Z

Is the branch up to date? Have you run the script, and pushed the code?

I still see a lot of wrong updates.

Last changes were not pushed yet, I will do after resolving few issues

vjgaur · 2024-11-18T06:52:25Z

Most importantly, cargo doc is failing. Are you building this yourself locally?

https://github.com/paritytech/polkadot-sdk/actions/runs/11757781965/job/32755463592?pr=6265

Hi @kianenigma,

I have been working on the cargo doc issue for frame_origin.rs. The script I am using should handle combining split references like:

// [`frame_runtime_types`]: crate::reference_docs::frame_runtime_types#
//! adding-further-constraints-to-runtime-composite-enums

into:

// [`frame_runtime_types`]: crate::reference_docs::frame_runtime_types#adding-further-constraints-to-runtime-composite-enums

However, even after my latest fixes, cargo doc is still showing the error:

error[E0753]: expected outer doc comment
   --> docs/sdk/src/reference_docs/frame_origin.rs:264:1
    |
264 | //! adding-further-constraints-to-runtime-composite-enums
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Could you help me understand if:

The approach of combining these split references is correct?
If there's something specific about this reference that needs different handling?

Here's the link to my current implementation: Last Commit

kianenigma · 2024-11-18T09:53:42Z

Your main issue seems to be that you are are prefixing lines with //, while everything should be //!.

Cargo doc will only parse //! lines :)

vjgaur · 2024-11-18T10:39:00Z

Your main issue seems to be that you are are prefixing lines with //, while everything should be //!.

Cargo doc will only parse //! lines :)

Thanks @kianenigma! You are right I completely misinterpreted the error message and went in the wrong direction. Will fix the script to keep //! as needed.

vjgaur · 2024-11-23T14:47:16Z

Your main issue seems to be that you are are prefixing lines with //, while everything should be //!.

Cargo doc will only parse //! lines :)

Hi @kianenigma I believe that issue is fixed .
Currently I see cargo doc build is throwing some Github action errors Can you please check ?
https://github.com/paritytech/polkadot-sdk/actions/runs/11987854773/job/33422527301?pr=6265

alvicsam · 2024-11-25T10:07:23Z

@vjgaur this is confusing CI error which I will fix. You should check the step above with the name Run forklift cargo test --doc --workspace. Also build-rustdoc is also failing: https://github.com/paritytech/polkadot-sdk/actions/runs/11987854773/job/33422527381?pr=6265#step:4:3420

PR addresses #6265 (comment) cc paritytech/ci_cd#1084

- Fixed //! inner doc comment issues in reference links - Properly handled anchor links on single lines - Removed duplicate reference entries - Maintained consistent // comment style in references - Fixed variable scope issues in link shortener script - Improved link processing and reference generation - Handled all link types (crate, frame, pallet) correctly

…-enums, reverted to previous code

vjgaur · 2024-11-28T09:10:46Z

@vjgaur this is confusing CI error which I will fix. You should check the step above with the name Run forklift cargo test --doc --workspace. Also build-rustdoc is also failing: https://github.com/paritytech/polkadot-sdk/actions/runs/11987854773/job/33422527381?pr=6265#step:4:3420

@alvicsam After running cargo test --doc --workspace , I am getting some weird error. Attaching the output screenshot. I tried all the fixes .
fatal error: 'algorithm' file not found
2 | #include

Also I see error log in build-rustdoc I have made no changes in those files and they are same as in the upstream master repo. My repo is fully upto date.

alvicsam · 2024-11-28T09:57:15Z

@vjgaur I can't help you with your local environment but you could run the command in our ci docker image.
The image name is: docker.io/paritytech/ci-unified:bullseye-1.81.0-2024-09-11-v202409111034. Also please pay attention that cargo test --doc --workspace should be run with RUSTFLAGS="-Cdebug-assertions=y -Dwarnings" env variable.

PR addresses paritytech#6265 (comment) cc https://github.com/paritytech/ci_cd/issues/1084

kianenigma reviewed Oct 28, 2024

View reviewed changes

kianenigma reviewed Nov 4, 2024

View reviewed changes

docs/sdk/src/external_resources.rs Outdated Show resolved Hide resolved

kianenigma reviewed Nov 5, 2024

View reviewed changes

kianenigma added the R0-silent Changes should not be mentioned in any release notes label Nov 5, 2024

kianenigma reviewed Nov 15, 2024

View reviewed changes

vjgaur requested review from a team as code owners November 23, 2024 14:16

alvicsam mentioned this pull request Nov 25, 2024

ci: improve workflow-stopper ux #6632

Merged

github-merge-queue bot pushed a commit that referenced this pull request Nov 25, 2024

ci: improve workflow-stopper ux (#6632)

c422d8b

PR addresses #6265 (comment) cc paritytech/ci_cd#1084

vjgaur added 10 commits November 28, 2024 13:43

docs: added script to shorten documentation links

dc4b7d5

docs: update link shortener script and apply changes based on feedback

d4f3287

minor change

2955794

Add PRDoc for documentation link changes

5957377

Added PRDoc for documentation link changes issue 6195

2a9ef0c

Added PRDoc for documentation link changes issue 6265 update

5baff24

Update PRDoc to use correct field names pr to number

8dc7b49

fixing pr check for prdoc

41a35ed

fixing pr check for prdoc

d8730a4

fix for pr_6265 prdoc

40cedf4

vjgaur added 9 commits November 28, 2024 13:43

removing crates since script isnt impacting any crates

e3d6b92

script update to last line prefixed

bd82ca7

removed Link References

5c82fc1

fixed cargo-clippy issues

42701a3

updated script fixing cargo clippy errors

879d0e5

fixed cargo doc issue adding-further-constraints-to-runtime-composite…

c1e0c02

…-enums, reverted to previous code

fix preserve //! in doc references

b64417a

upstream changes merged

f6e3188

vjgaur force-pushed the master branch from 057cdc7 to f6e3188 Compare November 28, 2024 08:17

Krayt78 pushed a commit to Krayt78/polkadot-sdk that referenced this pull request Dec 18, 2024

ci: improve workflow-stopper ux (paritytech#6632)

ff55eaf

PR addresses paritytech#6265 (comment) cc https://github.com/paritytech/ci_cd/issues/1084

dudo50 pushed a commit to paraspell-research/polkadot-sdk that referenced this pull request Jan 4, 2025

ci: improve workflow-stopper ux (paritytech#6632)

ebdf2a9

PR addresses paritytech#6265 (comment) cc https://github.com/paritytech/ci_cd/issues/1084

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: added script to shorten documentation links Fixes #6195 #6265

docs: added script to shorten documentation links Fixes #6195 #6265

vjgaur commented Oct 28, 2024 •

edited

Loading

kianenigma Oct 28, 2024

vjgaur Oct 29, 2024

kianenigma Oct 28, 2024

vjgaur Oct 29, 2024

vjgaur Nov 4, 2024

kianenigma Nov 5, 2024

vjgaur Nov 5, 2024

kianenigma left a comment

kianenigma Nov 5, 2024

vjgaur Nov 5, 2024

kianenigma Nov 15, 2024

kianenigma commented Nov 15, 2024

kianenigma commented Nov 15, 2024

vjgaur commented Nov 16, 2024

vjgaur commented Nov 16, 2024

vjgaur commented Nov 18, 2024

kianenigma commented Nov 18, 2024

vjgaur commented Nov 18, 2024

vjgaur commented Nov 23, 2024 •

edited

Loading

alvicsam commented Nov 25, 2024

vjgaur commented Nov 28, 2024 •

edited

Loading

alvicsam commented Nov 28, 2024

	//! [`async_backing_guide#timing-by-block-number`](crate::guides::async_backing_guide#timing-by-block-number).
	//! [`async_backing_guide#timing-by-block-number`].

	//! Once done, feel free to step up into one of our templates: [`templates`](crate::polkadot_sdk::templates).
	//! Once done, feel free to step up into one of our templates: [`templates`](crate::polkadot_sdk::templates).

docs: added script to shorten documentation links Fixes #6195 #6265

Are you sure you want to change the base?

docs: added script to shorten documentation links Fixes #6195 #6265

Conversation

vjgaur commented Oct 28, 2024 • edited Loading

Description

Integration

Preview changes:

Apply changes:

Review Notes

Approach

Results from Testing

Implementation details:

Checklist

Polkadot Address 👍

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma commented Nov 15, 2024

kianenigma commented Nov 15, 2024

vjgaur commented Nov 16, 2024

vjgaur commented Nov 16, 2024

vjgaur commented Nov 18, 2024

kianenigma commented Nov 18, 2024

vjgaur commented Nov 18, 2024

vjgaur commented Nov 23, 2024 • edited Loading

alvicsam commented Nov 25, 2024

vjgaur commented Nov 28, 2024 • edited Loading

alvicsam commented Nov 28, 2024

vjgaur commented Oct 28, 2024 •

edited

Loading

vjgaur commented Nov 23, 2024 •

edited

Loading

vjgaur commented Nov 28, 2024 •

edited

Loading