Standard field matching in apex code needs to account for variable data type

[pgonzaleznetwork]

The MetadataComponentDependency Tooling API object currently supports custom fields, so it is possible to see if a custom field is used in apex code.

Standard fields support is not available, but this library provides "where is this used" information for standard fields in validation rules, workflow rules/updates, etc. This works pretty well because we use the field unique name to find references i.e Lead.Industry is the "id" reference in field updates that use this standard field.

However, when it comes to apex code, it's very hard to tell if a standard field is actually being used.

What the library does at the moment is:

1. When a standard field reference is passed as the entry point, i.e Lead.Industry, we figure out the object name, in this case Lead
2. Then, we use the MetadataComponentDependency to find all metadata that references this object. This works nicely because if an apex class uses the Lead object in any way, the API will find it.

   sfdc-soup/lib/sfdc_apis/metadata-types/StandardField.js

   Line 156 in d784bc6

   async function queryDependeciesByObjectName(parentObjectName){

3. So now we have a list of classes that do something with the Lead object, now we need to manually inspect them to see if they actually use the field in question, which is the tricky part.

At the moment, all we do is we check if the body of the class contains any reference to the field name, i.e Industry. We do this by using a regular expression

sfdc-soup/lib/sfdc_apis/metadata-types/StandardField.js

Line 190 in d784bc6

async function searchFieldInCode(params){

This approach however is very naive because the apex class can use the word industry in hundreds of different contexts that are NOT the actual usage of the Lead.Industry field, such ass

String Industry; //random variable with the same name
SObjectField field = Account.Industry;// actually a field but WRONG object
IndustryValue//variable that matches on a part of the name
[SELECT Industry_Field__c FROM Lead];//correct object but wrong field, because the name matches
Account notALead = new Account();
notALead.Industry = gotcha//actually a field but wrong object again

The only valid references should be

SObjectField field = Lead.Industry;// direct reference to the sObject type
[SELECT Name,Industry, OtherField__c FROM Lead];//exact matching on soql for the correct object
Lead a = new Lead(Industry='Auto');
a.Industry = null;//a property/member of an object, where the object was instantiated with the correct object type

For this to work, we need some very robust/complex parsing logic. I'm open to using 3rd party libraries if that makes things easier. At a high level, the algorithm would have to be something like this:

1. Inspect the apex class line by line
2. On each line, see if there's an exact match of the object name, i.e lead but not myLead
3. If there's a match, use some logic to determine the type of match, i.e is this a comment, is it a variable being instantiated? if it is a variable, is the type lead? if so, does it have the field name in the constructor i.e Lead l = new Lead(Industry='cars')
4. Keep searching each line, if we find an exact reference to the field name i.e industry but not myIndustry we again need to ask the same questions? is this a comment, is it a variable? if it is a variable, does it belong to an object of that lead? If so, WE HAVE A MATCH!

[pgonzaleznetwork]

Another thing is that obviously, we are not the only app that has had similar requirements, so someone somewhere has already solved this problem and the approach should be: try to find an open-source library first, write our own parser only if really really necessary.

Here are some example libraries we could use to generate an AST from apex

https://www.npmjs.com/package/apex-parser
https://github.com/urish/java-ast
https://www.npmjs.com/package/java-parser
https://www.npmjs.com/package/java-method-parser

Part of the scope of this issue is to do some R&D on the above and figure out which would could satisfy our needs.

[pgonzaleznetwork]

Also we can use the SymbolTable object to determine the type of a variable!

[
  {
    "attributes": {
      "type": "ApexClass",
      "url": "/services/data/v51.0/tooling/sobjects/ApexClass/01p3h00000FHIq5AAH"
    },
    "Name": "SameName",
    "SymbolTable": {
      "constructors": [
        {
          "annotations": [],
          "location": {
            "column": 12,
            "line": 3
          },
          "modifiers": [
            "public"
          ],
          "name": "SameName",
          "parameters": [
            {
              "name": "std",
              "type": "ApexPages.StandardController"
            }
          ],
          "references": [],
          "type": null
        }
      ],
      "externalReferences": [],
      "id": "SameName",
      "innerClasses": [],
      "interfaces": [],
      "key": "SameName",
      "methods": [
        {
          "annotations": [],
          "location": {
            "column": 24,
            "line": 7
          },
          "modifiers": [
            "static",
            "public"
          ],
          "name": "doSomething",
          "parameters": [],
          "references": [],
          "returnType": "void",
          "type": null
        }
      ],
      "name": "SameName",
      "namespace": null,
      "parentClass": "",
      "properties": [],
      "tableDeclaration": {
        "annotations": [],
        "location": {
          "column": 14,
          "line": 1
        },
        "modifiers": [
          "public"
        ],
        "name": "SameName",
        "references": [],
        "type": "SameName"
      },
      "variables": [
        {
          "annotations": [],
          "location": {
            "column": 50,
            "line": 3
          },
          "modifiers": [],
          "name": "std",
          "references": [],
          "type": "ApexPages.StandardController"
        },
        {
          "annotations": [],
          "location": {
            "column": 14,
            "line": 8
          },
          "modifiers": [],
          "name": "l",
          "references": [],
          "type": "Lead"
        },
        {
          "annotations": [],
          "location": {
            "column": 16,
            "line": 9
          },
          "modifiers": [],
          "name": "realIndustry",
          "references": [],
          "type": "String"
        },
        {
          "annotations": [],
          "location": {
            "column": 17,
            "line": 11
          },
          "modifiers": [],
          "name": "Industry",
          "references": [],
          "type": "String"
        },
        {
          "annotations": [],
          "location": {
            "column": 23,
            "line": 12
          },
          "modifiers": [],
          "name": "field",
          "references": [],
          "type": "Schema.SObjectField"
        },
        {
          "annotations": [],
          "location": {
            "column": 19,
            "line": 15
          },
          "modifiers": [],
          "name": "notALead",
          "references": [],
          "type": "Account"
        },
        {
          "annotations": [],
          "location": {
            "column": 22,
            "line": 39
          },
          "modifiers": [],
          "name": "field2",
          "references": [],
          "type": "Schema.SObjectField"
        }
      ]
    }
  }
]