Reconfiguration stuck until internal queues emptied

[shlomos]

Hi,

I've been using FastClick with FromDPDKDevice and a processing custom element.
My current configuration performs a loopback of data as follows:
FromDpdkDevice(0) -> Processing -> ToDpdkDevice(0)
When I'm working with low rates at which the processing can be done on all packets, reconfiguration using hotconfig works well, however, if the rate is too high for all packets to be processed, the reconfiguration is very slow, especially for larger configurations.

Currently, we read 2048 bytes at a time, per a scheduler iteration, hence, on heavy load, when run_tasks takes a lot of time, the reconfiguration will be starving.

It seems that reading a larger chunk from the socket in controlsocket.cc:857 (or making this read dynamic according to the datalen read from the last command) solves this problem.

[tbarbette]

Hi Shlomo,

I've never really used reconfiguration, so it is kind of untested with DPDK. What do you change during the reconfiguration? Could you propose a simple configuration (maybe Processing being Null would be sufficient) with the steps to reproduce?

Are you sure controlsocket is the issue here? From what you say, I'd say FromDPDKDevice::run_task should be stopped (not call fast_reschedule) if a reconfiguration is pending. Simply setting _active to false during reconfiguration could solve it. Also, maybe Click's driver function should not continue through tasks if a reconfiguration is pending. But again I'm not very used to the reconfiguration process.

Thanks,
Tom

[shlomos]

So I guess a Classifier element with around 10,000 patterns each about 1000 bytes long should reproduce it. My current config is around 1.2MB (yea, I know it is an abomination...)

As for deactivating FromDPDKDevice, I really don't know. When processing packets at reconfiguration I assume one would like as little downtime as possible. Hence, keeping processing data is a positive thing. The reconfiguration however, is very inefficient in such cases as I have mentioned before.

I will analyze the downtime induced by deactivating the device at reconfiguration and report back as soon as I have a chance.

Cheers,
Shlomo

[tbarbette]

Unscheduling the tasks, making the changes, and rescheduling them could be as fast as running a timer. The only other option I see would be to have a specific check at the end of run_task. But then you pay a price at every loop... You cannot reconfigure while running run_task for sure. Eg if the number of queues change, the whole thing will just explode...

[tbarbette]

Any change on this?