better way of configuring xcom parameters?

[benmackenzie-exos]

just add task_id, parameter pairs to airflow.cfg, e.g.,:

[export_xcom]
abc = cdf
all = rec_count

then change xcom_cofig:

def load_xcom_config():
"""Loads the XCom config if present."""
if "export_xcom" in conf._sections:
return conf._sections['export_xcom']
else:
return {}

I would go ahead and do this and make a pull request, but am uncomfortable access conf._sections directly....conf.getsection('export_xcom') seems to be the right way to do this, but it won't work if the section is not one of the defaults. Looks like a bug to me...any ideas?

[abhishekray07]

I am not entirely sure I understand what you mean? Could you explain the problem you are trying to solve?

[benmackenzie-exos]

the problem is that in order to configure the xcom parameter exporter you need to go in an edit the config.yaml file. Which is a bit of a pain. It would be easier if the exporter got its configuration from the airflow.cfg file, which is accessible via conf._sections. My concern is that it seems wrong to access conf._sections directly. It seems like you should use conf.getsection...but this doesn't work.

[abhishekray07]

Yeah, I don't think adding anything to airflow.cfg would be appropriate since this exporter is installed as a plugin and this would create a weird circular dependency. Unfortunately, I don't have a good alternative either which might be easier to use. Ideally, a plugin like this just works out of the box without needing any to change any configurations or settings. That is the baseline for most people who use this plugin. If somebody needs more control, they can always go in and modify the configs.

[vanducng]

@benmackenzie-exos
I think you could change the directory storing config.yaml file to your AIRFLOW_HOME directory.
Simply make the modification to xcom_config.py

CONFIG_FILE = "/path/to/preferred/directory/config.yaml"