How can I hold or disconnect the telnet connection without error ?

[Tommylulu886943]

Question

Hello guys,
How can I hold or disconnect the telnet connection without error ?

Situation

I want to use the Nornir to reload the router, but when it be triggered the terminal will display the error messages "EOFError: telnet connection closed"

• When the reloading command be triggered, the router must be restart the system, so the telnet session will be disconnect

Action

I tried to use the r1.close_connections() after the reloading telnet conneciton, but it still triggered the EOFError before.

Execution Code

Environment

Python 3.9.6
nornir==3.1.1
nornir-jinja2==0.2.0
nornir-napalm==0.1.2
nornir-netmiko==0.1.1
nornir-rich==0.1.2
nornir-utils==0.1.2
netmiko==3.4.0
nornir-netmiko==0.1.1

Procedure

1. Reload the router
2. Wait for the reloading time of router
3. Show the router version

from time import sleep
from nornir import InitNornir
from nornir_netmiko import netmiko_send_command as send_cmd, netmiko_send_config

def reboot_device(task):
    task.run(
        task=netmiko_send_command,
        use_timing=True,
        command_string="reload",
    )
    task.run(
        task=netmiko_send_command,
        use_timing=True,
        command_string='y',
    )

def main():
    
    nr = InitNornir(config_file="config.yaml")
    r1 = nr.filter(device_name="R1")
    reload_time = 60

    r1.run(task=reboot_device)
    sleep(reload_time)
    r1.run(
        task=netmiko_send_command,
        use_timing=True,
        command_string="show version"
    )

if __name__ == "__main__":
    main()

Error Messages

2022-05-21 18:35:34,126 - nornir.core.task -    ERROR -      start() - Host 'R1': task 'netmiko_send_command' failed with traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/nornir/core/task.py", line 99, in start
    r = self.task(self, **self.params)
  File "/usr/local/lib/python3.9/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 30, in netmiko_send_command
    result = net_connect.send_command_timing(command_string, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/netmiko/utilities.py", line 500, in wrapper_decorator
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 1303, in send_command_timing
    new_data = self.read_until_pattern(pattern=re.escape(cmd))
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 655, in read_until_pattern
    return self._read_channel_expect(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 589, in _read_channel_expect
    output += self.read_channel()
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 526, in read_channel
    output = self._read_channel()
  File "/usr/local/lib/python3.9/site-packages/netmiko/base_connection.py", line 508, in _read_channel
    output = self.remote_conn.read_very_eager().decode("utf-8", "ignore")
  File "/usr/local/lib/python3.9/telnetlib.py", line 368, in read_very_eager
    return self.read_very_lazy()
  File "/usr/local/lib/python3.9/telnetlib.py", line 405, in read_very_lazy
    raise EOFError('telnet connection closed')
EOFError: telnet connection closed

Thanks a lot !

[dmulyalin]

Hi, have you tried using try/except blocks to catch the error within your task?

Alternatively have a look at RetryRunner - https://nornir-salt.readthedocs.io/en/latest/Runners/RetryRunner.html - it should handle your telnet issue case if you initiate it with run_reconnect_on_fail =True parameter.

[Tommylulu886943]

Yes, I have tried to using try/except to catch the error like below, but it still returned the EOFError: telnet connection closed

try/except

    try:
        r1.run(task=reboot_device)
    except Exception:
        pass

I'll use RetryRunner to try again. Thank you for your time and help !!

[Tommylulu886943]

I have tried the RetryRunner with run_reconnect_on_fail=True on my case, it also will triggered the EOFError: telnet connection closed few times.

RetryRunner Code

    def task_manually_conneciton(task):
        try:
            task.run(
                task=netmiko_send_command,
                use_timing=True,
                command_string="reload",
            )
            task.run(
                task=netmiko_send_command,
                use_timing=True,
                command_string='y',
                cmd_verify="False",
            )
        except Exception:
            pass

    nr = InitNornir(
        runner={
            "plugin": "RetryRunner",
            "options": {
                "num_workers": 100,
                "num_connectors": 10,
                "connect_retry": 3,
                "connect_backoff": 40000,
                "connect_splay": 100,
                "task_retry": 3,
                "task_backoff": 1000,
                "task_splay": 100,
            }
        },
        config_file="config.yaml"
    )        

    r1 = nr.filter(device_name="R1")
    try:
        r = r1.run(
            task=task_manually_conneciton,
            run_reconnect_on_fail=True,
        )
    except Exception:
        pass

    if __name__ == "__main__":
        main()

I think after the singal task send the command "y" to the router, the router will reload the system, so it won't response the netmiko process and raise the error.

Could I have any method to avoid the disconnection error ? or Could I use any parma to trigger the disconnect after the command is send ?

[dbarrosop]

You should encode what you expect. If you expect an EOFError when you close the connection you should close the connection yourself with a a try...except EOFError block so you can verify you are getting the error you expect.

[Tommylulu886943]

@dbarrosop Thank you for your help. I have tried to use try...except EOFError to handle the error, and finally to do the .close_connections()
But I think before the .close_connections() operation, when the "y" command send that the router immediately disconnect the connection.
I want the reloading event, that won't raise the EOFError.

    r1 = nr.filter(device_name="R1")
    try:
        r = r1.run(
            task=task_manually_conneciton,
        )
    except EOFError:
        r1.close_connections()

[Tommylulu886943]

I figure out the problem, before I executed the .send_command('y') the telnet connection is disconnect, in my route, after enter the reload command if received the \n then will immediately reload the system, so it triggered the error after the send the .send_command('y')

from nornir import InitNornir
from nornir.core.task import Task, Result 
from nornir_netmiko.tasks.netmiko_send_command import netmiko_send_command

def main():
    
    def task_reboot(task):

      task.run(
          task=netmiko_send_command,
          use_timing=True,
          command_string="reload",
       )
       task.host.close_connections()

    nr = InitNornir(config_file="config.yaml")        
    r1 = nr.filter(device_name="R1")

    try:
        r = r1.run(
            task=task_reboot,
        )
    except EOFError:
        r1.close_connections()
        
if __name__ == "__main__":
    main()