[CT-73] [Feature] Better error messages: include line number and location for Compilation Error

[tekumara]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

When encountering a Compilation error it's not clear which line the error is on, eg:

.venv/bin/dbt run-operation drop_dev_schema
06:36:30  Running with dbt=1.0.1
06:36:30  Unable to do partial parsing because a project config has changed
06:36:33  Encountered an error:
Compilation Error in model lineitem (models/lineitem/lineitem.sql)
  'NoneType' object is not iterable
  
  > in macro generate_sat_table (macros/generate_sat_table.sql)
  > called by model lineitem (models/lineitem/lineitem.sql)

Describe alternatives you've considered

No response

Who will this benefit?

Anyone trying to debug compilation errors

Are you interested in contributing this feature?

Yes

Anything else?

dbt 1.0.1

[nathaniel-may]

Thanks for the feature request @tekumara! I think the addition of line numbers would be a big help debugging dbt projects, and I couldn't find any previous feature requests for this.

I do want to point out that the particular error you're running into is an internal Python error that should never be raised to users. If you have a clear minimal reproduction case for that, it would be very helpful if you could log a bug ticket for what causes 'NoneType' object is not iterable to be raised.

However, with other compilation errors such as models depending on models that do not exist (typo in model name) or calling an undefined macro (typo in macro name), line numbers are still not included today.

I see that you're interested in contributing to this feature, which we greatly appreciate. To implement this I believe we would have to preserve line numbers when we read files, and check them on every exception that we raise. This isn't exactly a small lift, and we have tentative plans to centralize exceptions to the event module which may help prepare for this kind of effort.

[nathaniel-may]

Talking to the team, this might be a bit more complicated to get line numbers because a lot of this work is done in batches by jinja where we won't be able to preserve line-level granularity. However, we might be able to point out the location of the jinja blocks that contain errors when jinja raises them.

If you decide to open up a PR please feel free to tag me as a reviewer.

[jtcohen6]

To provide a concrete/motivating example, if you include the following code in a model:

-- models/my_model.sql
{% set mylist = None %}

{% for item in mylist %}

  select '{{ item }}' as my_item
  {{ 'union all' if not loop.last }}

{% endfor %}

You'll see the reported error:

$ dbt run
14:49:59  Running with dbt=1.0.0
14:49:59  Encountered an error:
'NoneType' object is not iterable

In this case, it's with even fewer helpful pointers — not even a mention that the error was encountered while parsing my_model.sql.

[nathaniel-may]

I just spoke with @jtcohen6 about this, and for anyone who wants to try and get this working in a PR, this is where many of the jinja blocks end up being rendered, and there's an existing context manager that catches jinja rendering exceptions to raise better dbt exceptions.

Additionally, instead of threading line numbers through, we could just return the text of the offending jinja block which might be an easier lift.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[jaypeedevlin]

This is on my list of things to get to (cc @ehmartens) so let's leave it open!

[markfickett]

I also spent a few hours tracking down the source of a 'NoneType' object is not iterable error w/ no trace (dbt 1.6.0). This was from a custom macro that fetches some inputs via dbt_utils.get_column_values. A detailed error message would be a huge improvement!

The dbt exceptions.raise_compiler_error method was very useful for debugging (and useful to leave in as a precondition check), since jinja doesn't have standard Python exception/assert mechanisms. Here's a basic example in case others find it a useful workaround:

{% if my_list == None %}
  {{ exceptions.raise_compiler_error("Invalid `my_list`. Got: " ~ my_list) }}
{% endif %}

[ashwini-kumar2]

I'm getting

07:10:16 Running with dbt=1.7.13 07:10:21 Registered adapter: databricks=1.7.13 07:10:21 Unable to do partial parsing because of a version mismatch 07:10:24 Encountered an error: Compilation Error Encountered unknown tag 'endmacro'. You probably made a nesting mistake. Jinja is expecting this tag, but currently looking for 'endfor' or 'else'. The innermost block that needs to be closed is 'for'. line 105 {%- endmacro -%}

Its very difficult which macro is breaking at what point

[devmessias]

I think it might be better to first discuss capturing a TypeError during the Jinja render phase (this is what I'm requesting here: #10856)), based on the example provided by @jtcohen6. After that, address the line numbers info in the exception message