Automated 404 link removal

[meganthecoder]

@beekerr is interested in building an automated job that will edit content and remove links on the page that 404 after a certain amount of time (say, 6 weeks). The idea is to mimic the functionality that exists in AEM.

I was thinking that the job would either need to be build in Adobe I/O or on the Franklin side, coordinating with the Franklin team. Are there any other ideas here?

Also, I'm thinking he'd need to work with the sharepoint api to get/update content. It seems like we might run into issues with authors actively editing a doc or if a doc has unpublished changes.

Does the community have any advice or ideas on how to accomplish this?

[meganthecoder]

@sunilkamat @arshadparwaiz @sirivuri It seems like floodgate is using some similar tools. Any ideas?

[sukamat]

@meganthecoder Floodgate is not modifying any docx file. The regional-edits module in loc is.

In AEM, the published files were on a separate "publish" server and the flattened 404 links were not really affecting the content being updated in "author" instance.

In Milo, as there is just one source of the content (docx), even if you fix the 404 links, you cannot publish them as there might be other edits in that doc that should not be published yet.

Generating a report that lists all the pages that have invalid links might be a good first step which does not really have any dependency on modifying the docx files or worrying about the state that it is in. Also that report can be generated just by parsing the end-user facing HTML pages instead of pinging Sharepoint and modifying docx files which will be a heavy operation. May be the Milo Automation team already have this report/tool?

Also reach out to the Franklin team to see if they have any tools to flatten the 404 links before they cache the pages.

[meganthecoder]

@sukamat Thanks for the context. Those are good callouts and dealing with non-published changes is definitely a concern. Do you think that editing the doc isn't a viable option?

@beekerr said they have a reporting tool already. Do you have a link/repo for us to look at?

FYI, I've reached out to the Franklin team too, and I'm waiting for a response.

[meganthecoder]

This was @beekerr 's response via slack about link reporting tools:

yeah we have a few, our main source is https://www.lumar.io/ (3rd party large scale crawler) and we have a few internal tools that we are developing. no repo to share yet but here is the main wiki https://wiki.corp.adobe.com/display/seoteam/SEO+Guardian+PRD

that aside, we have the ability to determine when a link should be decommissioned, we just dont have the ability to fix it in automated fashion

[auniverseaway]

I don't understand the request. Will try to re-phrase based on my understanding.

"As an SEO expert, I would like the ability to automatically remove links that live in Word documents that may point to 404 pages."

Is this right?

[meganthecoder]

That's good clarification. I believe the request is to remove the 404 links from the page, and I've added that we'd want to do this on the word doc. I assumed that doing this on the frontend would be too costly for performance, but we can have that discussion too. Maybe this is more accurate:

"As an SEO expert, I would like the ability to automatically remove 404 links (links that point to 404 pages) from the content of all pages on the site."

[meganthecoder]

@auniverseaway @sukamat Thanks for your input! I'm trying to write up some conclusions for this due diligence. Here are the possibilities I've evaluated. Please let me know if you have thoughts

Frontend solution - Bad User Experience
This would entail fetching every link on a page on load and processing them. It would have a bad impact on performance, which goes against the Milo philosophy.

Pre-cache solution - Dependent on Franklin Team
This would entail processing links and removing bad links before the page is cached on the Franklin side to reduce the performance impact. It would need buy-in from the Franklin team, which we don't currently have.

Automated authoring solution - Complex and Risky
Editing the actual word doc (content source) and publishing those changes would come with major challenges. For example, what if there are unpublished changes in the document? You would have to have a way to detect that and skip it or you'd publish changes that shouldn't be public. All of our environments share a single content source, so making automated edits would be risky and could create conflicts with authors. It would also be technically challenging, as we've seen with other projects that use the sharepoint api.

Manual authoring aides - Do-able
There are several things we could do to help the manual authoring workflow. First, we could make enhancements to the Preflight link validator. We'd want to know what would be helpful here from an authoring perspective, but some possibilities could be: listing the number of bad links, their types or the links themselves, etc. Also, if an internal reporting tool is built, we could add more information to it than an external tool, such as the last person to edit the page, etc.

[meganthecoder]

@sirivuri You're working on a scheduled script in New Relic that sends emails, right? Do you think that could be useful in this case to let authors know about bad links on their pages?

[fullcolorcoder]

Have you considered creating a middle solution similar to what's happening placeholders.json?
Set up an intake form/excel for 404s found in link checkers that log the invalid link / query selector / page info, and then use it to invalidate the link the same way we do placeholders.

Now the links are handled, authors are notified and we don't touch docs.

[sukamat]

@meganthecoder Showing the link errors in pre-flight is a great idea. That will fix the source.

Given that there is already a reporting tool, the approach that @fullcolorcoder mentioned should work as well. If the tool that generates the report provides precise information that can be added to an excel and loaded as a JSON, perhaps the links could be flattened that way. It might be good to split that out by regions to avoid having a massive JSON listing all the invalid links across the site. Worth a POC to check on performance impact.

The process should still have a way, either manual or automated, to notify the content admin or doc owner so that they can go fix the source docs.

[meganthecoder]

@fullcolorcoder That's an interesting idea. I got the sense that it was a large volume of links, but I can double check on that. It would be another thing happening before decorating content, which isn't ideal, but we can always do a POC like @sukamat said to see the true impact. I'll follow up on some of those questions with @beekerr

[arshadparwaiz]

@sirivuri You're working on a scheduled script in New Relic that sends emails, right? Do you think that could be useful in this case to let authors know about bad links on their pages?

@meganthecoder the New Relic script we are working on is to test a single external API availability and expected response. New Relic supports Node.js script, which can include any logic, this implementation is more aligned towards sanity checks for Live systems/APIs. Technically the 404 solution is doable in the New Relic script, however may not be a suitable for the problem we are solving for this. Let us know if any details are required to further explore this option.
cc: @sirivuri