Convert delay_factor/global_delay_factor to a simple read_timeout for the send_command method

[ktbyers]

I have been thinking quite a bit about how global_delay_factor and delay_factor should be fixed. For send_command , I was planning on doing the following:

1. Completely kill both delay_factor and global_delay_factor here i.e. as of netmiko4, these would do nothing on the send_command() method (probably just raise a warning if you try to use them).
2. Add a read_timeout that would be an argument to send_command. For example, send_command("show ip int brief", read_timeout=30). read_timeout will be the time Netmiko spends looking for the terminating pattern either based on the prompt or expect_string if it is specified. read_timeout will default to 10 seconds.
3. If terminating pattern (i.e. prompt or expect_string) is not detected in read_timeout seconds, then ReadTimeout exception will be raised.

Because of platforms like Ansible, Salt, Nornir and also libraries like NAPALM which might potentially make argument passing to a method difficult, there will be a ConnectHandler argument that will override the default read_timeout (probably read_timeout_override=x).

tldr = I am planning on killing delay_factor/global_delay_factor to send_command and replacing it with a simple read_timeout=x

Thoughts/issues/problems?

[EndlessDynamics]

Kirk, I really like the idea. It’s more user-friendly as it translates to a concept one would think of when the desire is “keep trying until _____”.

…

________________________________ From: Kirk Byers ***@***.***> Sent: Tuesday, June 8, 2021 12:42 To: ktbyers/netmiko Cc: Subscribed Subject: [ktbyers/netmiko] Convert delay_factor/global_delay_factor to a simple read_timeout for the send_command method (#2302) I have been thinking quite a bit about how global_delay_factor and delay_factor should be fixed. For send_command , I was planning on doing the following: 1. Completely kill both delay_factor and global_delay_factor here i.e. as of netmiko4, these would do nothing on the send_command() method (probably just raise a warning if you try to use them). 2. Add a read_timeout that would be an argument to send_command. For example, send_command("show ip int brief", read_timeout=30). read_timeout will be the time Netmiko spends looking for the terminating pattern either self.base_prompt or expect_string if it is specified. read_timeout will default to 10 seconds. 3. If pattern (i.e. self.base_prompt or expect_string) is not detected in read_timeout seconds, then ReadTimeout exception will be raised. Because of platforms like Ansible, Salt, Nornir and also libraries like NAPALM which might potentially make argument passing to a method difficult, there will be a ConnectHandler argument that will override the default read_timeout (probably read_timeout_override=x). tldr = I am planning on killing delay_factor/global_delay_factor to send_command and replacing it with a simple read_timeout=x Thoughts/issues/problems? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#2302>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMUHHMWJXNUULJVE6ILAV6TTRZB6LANCNFSM46KKLUBA>.

[dmulyalin]

But what if device continue sending the output and read time out expire, think we should have two timeouts, one is absolute, which works as you describing read timeout now and second timeout that should expire if no output from device was received for certain amount of time. All in all, that was implemented with promptless mode PR I raised a while ago.

[dmulyalin]

Would say 120s for absolute time out and 30s for read time out should cut a decent amount of use cases.

[dmulyalin]

Also, having two separate methods - send_command() and send_command_timing() - is rather questionable to me and definitely adds complexity, having absolute/relative timers introduced in pattern based send_command method would render send_command_timing useless, as:

• if no pattern detected in output it will terminate based on the timeout rendering send_command_timing behavior
• to simulate send_command_timing behavior one can set pattern to never match string e.g. __bogus_1234_bla__

Think that merging send_command() and send_command_timing() in a single method would decrease complexity for end user.

[ktbyers]

Yeah, that breaks way too much existing end-user code.

Also entirely timing based method is good for situations that prompt for additional information (working out all the prompts can be painful and time consuming). Which is a pretty common pattern.

You wouldn't simulate send_command_timing by setting the pattern to a bogus string. In order to simulate send_command_timing() you would have to: 1)significantly reduce the timer(s), 2)set a bogus pattern, 3)handle the exception that was generated by the timers expiring.

You could simulate it by setting pattern to a null string, but then you would still have to adjust the timer(s). It would still be a lot easier just to have a timing based method.

[iskobkarev]

Well this is a great conversation with all the right people involved!

I've been trying to pull a relatively long BGP tables from hpe/cisco devices and parse them with ttp. The read often fails upon timeout, no matter how large I make it.

Would it make it easier to reset the timeout if the data is still being received and start counting only when the data stream stops and keep the timeout relatively short?

Some absolute timeout would probably be necessary for some odd cases but that approach should address the majority of read situations where a short output and the long output would be covered with the same relatively short timeout

[ktbyers]

Are you using Netmiko 4.0.0?

If so, then you should be able to just use send_command(command, read_timeout=x) where x is a time in seconds sufficiently large that your output will complete. read_timeout is an absolute timer (that is very unlikely to change).

[iskobkarev]

yes I installed 4.0 literally a few days back I'll give a try in a bit - thank you What I am talking about is a somewhat different concept of starting the timer when there is no longer data coming from the device and the prompt is still not matched). I understand there could be a few cases: - data arrived and then the prompt -> all is good here - prompt has arrived but some junk/background data is still arriving -> then the prompt should be matched anyway - data stopped but no prompt -> then timer kicks in when the last bit of data arrived and expires - no prompt / no data -> the timer kicks in when the command was sent and expires

…

On Wed, Mar 30, 2022 at 4:56 PM Kirk Byers ***@***.***> wrote: Are you using Netmiko 4.0.0? If so, then you should be able to just use send_command(command, read_timeout=x) where x is a time in seconds sufficiently large that your output will complete. read_timeout is an absolute timer (that is very unlikely to change). — Reply to this email directly, view it on GitHub <#2302 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATJNEU6UCWL5RKGSA5HPKYLVCTSZ3ANCNFSM46KKLUBA> . You are receiving this because you commented.Message ID: ***@***.***>

-- Thanks, Igor Skobkarev, ***@***.***

[ktbyers]

@iskobkarev All of the above would be properly handled by send_command() and/or send_command_timing()?

• data arrived and then the prompt -> all is good here

Standard behavior.

• prompt has arrived but some junk/background data is still arriving ->
  then the prompt should be matched anyway

Something arriving after the prompt would likely have to be unrelated to the specific show command output (so something like logging messages, or other alerts also being sent to the vty). So we capture the output we intended to capture i.e. the show command.

• data stopped but no prompt -> then timer kicks in when the last bit of
  data arrived and expires

No prompt ever coming would raise a ReadTimeout exception which you could control the length of. You can also control the terminating pattern using the expect_string argument. I guess you would lose partial data since you would get an exception and never receive what had occurred up until that point. But if there is never any terminating pattern you can match on then you should switch to send_command_timing() which should handle this case.

• no prompt / no data -> the timer kicks in when the command was sent and
  expires

Nothing ever coming back it would raise a ReadTimeout exception after read_timeout time. You could try/except this if you wanted to gracefully handle it. Of course, your CLI would be dead until the prompt came back (or until you sent some sort of break sequence or cntl-c down the channel).

I guess I am not really seeing a case that is a problem? The most problematic one is case 3, but it seems like Netmiko could handle that case. Do you have an example command that might fall into case 3?

[iskobkarev]

re: I guess I am not really seeing a case that is a problem

Agreed as I did not indicate case #5 for some reason -- something must have distracted me:

• the proper response data keeps arriving for the duration above read_timeout and no prompt yet.

In my case it was HPE's Comware 7 "display bgp l2vpn evpn" which produces a lengthy and rather slow output which (sometimes) exceeds 200 seconds read timeout I tried to use with 4.0. The output for that command when redirected to a local file on the switch takes ~30 seconds and produces ~12 Mb output. A ssh client would take ~60 seconds (depending on the network) and with netmiko it takes 200 sec or more quite often.

thanks

[ktbyers]

@iskobkarev So you are seeing 200 seconds for Netmiko when it is only 60 seconds for the SSH client? I don't see why that would be on the face of it, Netmiko should be essentially the same here (i.e. 60 seconds in both cases).

That would definitely be worth getting to the bottom of.

In other words, there should be no meaningful difference in the actual read time of Netmiko as compared to a manual SSH session.

[zohassadar]

After digging into send_command to deal with a corner case recently, I came across this thread and figured I'd throw my opinion into the mix. I like the idea of having two timeouts. Something short to account for the the device that's gone quiet and something longer to account for the output that will never stop.

Here's an idea to throw out there

start_time = last_read_time = now = time.time()

Two additional variables are set to time.time() along with start_time

start_time remains untouched
last_read_time is updated every time data is received
now is updated every loop

now - last_read_time < read_timeout and now - start_time < read_timeout * 10

This while loop condition will fail if no data has been received before the read_timeout, or if data is still being received and 10x the read_timeout has passed.

Test code:

import os
import time

import netmiko

def send_command_with_timeout(command: str, read_timeout: int):
    session = netmiko.ConnectHandler(
        host="192.168.42.254",
        device_type="cisco_ios",
        username=os.getenv("USERNAME"),
        password=os.getenv("PASSWORD"),
    )
    try:
        error = None
        start = time.time()
        session.send_command(command, read_timeout=read_timeout)
    except netmiko.ReadTimeout:
        error = 'timeout'
    finally:
        finish = time.time()
    print(f"{command=} {read_timeout=}  execution={finish-start:.02f} {error=}")

In all examples, the read_timeout was set to 2 seconds. The 10x multiplier means the absolute timeout is 20 seconds.

Example command exceeding read_timeout:

>>> send_command_with_timeout("show tech", 2)
command='show tech' read_timeout=2  execution=3.96 error=None

Another example exceeding read_timeout:

>>> send_command_with_timeout("ping 192.168.42.1 size 18024 repeat 1000", 2)
command='ping 192.168.42.1 size 18024 repeat 1000' read_timeout=2  execution=15.45 error=None

Example of expected prompt that won't be found:

>>> send_command_with_timeout("configure", 2)
command='configure' read_timeout=2  execution=2.11 error='timeout'

Example output exceeding absolute timer:

>>> send_command_with_timeout("ping 192.168.42.1 size 18024 repeat 2147483647", 2)
command='ping 192.168.42.1 size 18024 repeat 2147483647' read_timeout=2  execution=20.08 error='timeout'

[ktbyers]

@zohassadar I am trying to parse what you are saying/asking for here...it seems like the main thing you are asking for is a quick failure in the case where it goes quiet i.e. you have some output and then no more output (and the prompt or expect_string is not matched).

The other three cases would already be handled by just setting read_timeout=20.

So in the above that would be this case here:

Example of expected prompt that won't be found:

>>> send_command_with_timeout("configure", 2)
command='configure' read_timeout=2  execution=2.11 error='timeout'

And in the current Netmiko read_timeout solution, you would do that as follows:

conn.send_command("configure")

And you would get an exception in 10 seconds (as that is the default) or 20 seconds if you set read_timeout=20

Am I reading this correctly?

[zohassadar]

You are.

To add context, there's a specific experience years ago where I had a script that had to accommodate a lot of devices, some being nexuses stacked with fexes. Commands like show interface on those would sometimes take over 2 minutes to complete. Because of this I had the delay_factor and max_loops set to something that caused about a 3 minute timeout. An instance came up where I had to debug a specific device and during a period when I expected a large amount of data to be coming in, the device was hanging out at the --More-- prompt. I don't remember the cause, but for this one device something caused the automatic sending of terminal length 0 to either get overridden or not acknowledged by the device. The solution was to add a redundant terminal length 0 to the script. Before figuring out why, I had spent quite a bit of time waiting. Now admittedly had I gone to the debugs much sooner than I did, I would have found out what was going on right away, but coding was harder then. That experience made me want to have something that would make the script die sooner if the device was quiet, and later if the device was still talking.

Fast forward to this week where I wasn't even aware of 4.0.0 until it introduced an unrelated issue and it gave me a chance to dig into the code. The read_timeout logic is solid and I like how simple and easy to understand it is. It looked like it would take hardly any modification at all to have two timeouts. The functional proof of concept referenced above showed that to indeed be the case.

The example commands above, and the timers used, were only to demonstrate the different behavior depending on whether or not the device was producing output when the lower timer was reached. For a real world scenario, I would generally expect the device to not go quiet for any more than 10 seconds at a time and that any command should complete in under 100 seconds.

Consider a user facing application that needs to collect show run from a device of the user's choosing. When everything is working, the command completes in well under 10 seconds. Then the first unexpected use case pops up. The user, vpn'd through their cell phone, chose a stack of 9 switches configured with dot1x on every port. After investigating it becomes apparent that the timeout needs to be at least 10 minutes.

There's a few ways that one could approach the problem, the simplest of which is to raise the value of read_timeout when calling send_command to 600 or higher. This would work great for guaranteeing that the function has enough time to receive all available data in all cases, but it doesn't make for a very good user experience when the 10 minutes are spent waiting for data that will never come. While rare, situations do crop up like the aforementioned terminal length 0 issue. There's also the case of new software/hardware coming out that may return an unexpected prompt. Then of course there's gremlins making the router do stupid things on occasion. In all of those odd cases, I'd prefer the user not have to wait 10 minutes to be told there's a problem. I also like the idea of knowing that if there is a wait, it's because data is coming.

I think the best solution to prevent unnecessary wait time and to also account for long output is to have two timers, one relative to when data was last seen and the other absolute to guarantee the script ends eventually. That would allow for a 10+ minute upper limit in the example above while still alerting the user quickly when a device goes quiet unexpectedly. I acknowledge that my use cases and considerations may not apply to all, but I wanted to throw it out there anyway. I now know how to implement the feature that I've been looking for and I'm perfectly alright rolling my own solution.

[dmulyalin]

Just to clarify - but it doesn't make for a very good user experience when the 10 minutes are spent waiting for data that will never come - netmiko will not wait for 10 minutes unconditionally and will return output as soon as it detects the prompt, raising read_timeout to 10 minutes will work for fast devices with no issues, but will allow slow devices or big output to accumulate over 10 minutes until prompt seen.

I think the best solution to prevent unnecessary wait time and to also account for long output is to have two timers - agree and it was implemented in promptless mode as a Nornir task.

There's also the case of new software/hardware coming out that may return an unexpected prompt. - netmiko_send_command_ps can handle changin prompt as well, have a look at the code if you willing to use it in your own solution.

the device was hanging out at the --More-- prompt - netmiko_send_command_ps can handle pagination as well as it sends two spaces by default to verify end of output

[ktbyers]

@zohassadar Let me read your response above again. I read it earlier, but want to re-read it a second time.

[ktbyers]

@zohassadar Okay, I read through your comment above again. I am somewhat open to making a new method for Netmiko 5.0 something like send_command_incr which has an incremental timeout.

So send_command would stay as it is, but there would be a new method that had an incremental timeout (and probably an optional absolute timeout that would default to being disabled, but that you could enable if you so chose).

I don't think I would want to change send_command again, but this would provide end-users with both choices.

The most common use case I can think of is the lots of devices use case where you are forced to large timeouts with send_command due to trying to accommodate a large number of devices and a large number of show commands so this forces you to larger and larger timeouts (which then forces you to larger waits on error). Though this is somewhat mitigated by you almost always using concurrency in these situations.

[dmulyalin]

@ktbyers all in all think it would be for the best of everybody to consider implementing plugins system for Netmiko 5 where certain functionality, including read from prompt mechanism, could be implemented as plugins similar to how Nornir does it using entry points.

That would allow to lift the burden off the Netmiko maintainers to test and support corner case scenarios and will allow people to use their own plugins.

[ktbyers]

@dmulyalin I will consider plugins for Netmiko 5. I actually didn't really like the plugin system for Nornir; I think it made Nornir worse, but was totally understandable as maintenance burden was/is a significant problem.

Ultimately, anyone can do what they want as far as constructing send_command patterns as they can just access the low-level read_channel() and write_channel() methods.