register workers of scheduler are less than workers in dashborad

[FANLONGFANLONG]

What happened:

• we used the below way to start dask cluster with 20 workers

   n_workers = 20
    spec_worker = skein.Service(resources=skein.Resources(memory=memory_limit, vcores=threads_per_worker),
                                instances=n_workers,
                                files=resource_files,
                                script=worker_script)
    services["dask.worker"] = spec_worker
   spec_scheduler = skein.Service(resources=skein.Resources(memory="20g", vcores=4),
                                       files=resource_files,
                                       instances=1,
                                       script=scheduler_script)
    services["dask.scheduler"] = spec_scheduler

    spec = skein.ApplicationSpec(name=app_name,
                                 queue=_get_queue(),
                                 master=spec_master,
                                 services=services)

    cluster = YarnCluster.from_specification(spec)

• scheduler log showed only few worker registered
  
• but we can see all workers in dashborad
  

What you expected to happen:
all of 20 workers registered into scheduler

Anything else we need to know?:

• sometime scheduler showed all of 20 workers registered, but sometime not.
• submitted task always ran into issue as below when scheduler showed only part of 20 workers registered
  

Environment:

• Dask version: 2.19.0
• Python version: 3.6.7
• Operating System: CentOS Linux release 7.2.1511 (Core)
• Install method (conda, pip, source): conda

[jacobtomlinson]

This seems to be related to how dask-yarn exposes scheduler logging so moving the issue over there.