Fixes for shapely and scipy updates #174

mdpiper · 2024-10-10T17:31:49Z

This PR includes two fixes to pymt for updates to shapely and scipy.

In shapely, the asShape adapters have been removed. The migration guide recommends replacing these adapters (such as asPoint) with geometries (Point). This fixes #164.

In scipy, a bug was identified with the next and previous methods when extrapolating above and below a range, respectively. The expected behavior is for these methods to fail in these situations. This fixes #162.

With these fixes, the version limit on these packages has been removed. I also reverted the "Don't install components for testing" change which was failing because of #164.

The subsequent call to `mamba install` updates *shapely* to 2.0, which breaks the <1.9 requirement from #164.

See https://shapely.readthedocs.io/en/stable/migration.html#other-deprecated-functionality

See scipy/scipy#16997 (comment)

This reverts commit bfb47fc. The shapely version issue is resolved.

See #173.

Without this setting, the value of legacy=1.25 was being inherited from Landlab, which isn't allowed in the installed version of numpy<2 (currently 1.26.4).

mdpiper · 2024-10-10T18:05:15Z

I required numpy<2 for older components. This can be lifted after the components are updated.

On top of this, I added np.set_printoptions(legacy="1.21") to override the default (1.25) that was coming from Landlab, and which isn't allowed in the currently installed numpy, 1.26.4.

mdpiper · 2024-10-10T18:24:21Z

The seven failing tests on Windows are emitted from Landlab from the same Cython file. Here's the last few lines of one of the stack traces:

  ...
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\utils\jaggedarray.py", line 115, in unravel
    unravel(data, offset, out)
  File "src\\landlab\\utils\\ext\\jaggedarray.pyx", line 28, in landlab.utils.ext.jaggedarray.unravel
ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'long'

and here's the full stack trace:

_____________________ test_structured_quadrilateral_grid ______________________
Traceback (most recent call last):
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\runner.py", line 341, in from_call
    result: TResult | None = func()
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\logging.py", line 829, in _runtest_for
    yield
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\capture.py", line 880, in pytest_runtest_call
    return (yield)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\pluggy\_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\_pytest\python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "D:\a\pymt\pymt\tests\framework\test_bmi_ugrid.py", line 136, in test_structured_quadrilateral_grid
    grid = StructuredQuadrilateral(bmi, grid_id)
  File "D:\a\pymt\pymt\pymt\framework\bmi_ugrid.py", line 206, in __init__
    graph = StructuredQuadGraph(nodes, shape=tuple(shape))
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\graph\structured_quad\structured_quad.py", line 633, in __init__
    StructuredQuadGraphExtras.__init__(self, (node_y, node_x), sort=sort)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\graph\structured_quad\structured_quad.py", line 508, in __init__
    Graph.__init__(
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\graph\graph.py", line 857, in __init__
    self._ds = ugrid_from_unstructured(node_y_and_x, links=links, patches=patches)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\graph\ugrid.py", line 28, in ugrid_from_unstructured
    _update_links_at_patch(ugrid, patches)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\graph\ugrid.py", line 74, in _update_links_at_patch
    patch_links = links_at_patch(patches)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\graph\matrix\at_patch.py", line 40, in links_at_patch
    return unravel(links_at_patch, offset_to_patch, pad=-1)
  File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\utils\jaggedarray.py", line 115, in unravel
    unravel(data, offset, out)
  File "src\\landlab\\utils\\ext\\jaggedarray.pyx", line 28, in landlab.utils.ext.jaggedarray.unravel
ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'long'

mcflugen · 2024-10-10T18:34:40Z

...
File "C:\Users\runneradmin\miniconda3\envs\test\lib\site-packages\landlab\utils\jaggedarray.py", line 115, in unravel
unravel(data, offset, out)
File "src\landlab\utils\ext\jaggedarray.pyx", line 28, in landlab.utils.ext.jaggedarray.unravel
ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'long'

That's interesting, I would have thought the Landlab unit tests would have picked this up. I think it should be an easy fix.

mdpiper · 2024-10-10T18:52:34Z

Added Python 3.11 to the tests. We can't go to 3.12 because some older components aren't there yet.

mdpiper · 2024-10-10T20:20:23Z

This is off-topic from the rest of the PR, but because Mambaforge is going away quickly, I did a quick health check on the CI workflows.

mdpiper · 2024-10-10T20:31:21Z

@mcflugen What do you think we should do? It seems like everything is OK except for the Windows tests which are failing because of an upstream issue.

mcflugen · 2024-10-10T21:04:19Z

@mcflugen What do you think we should do? It seems like everything is OK except for the Windows tests which are failing because of an upstream issue.

I think we need a new release of Landlab. I'll have a look but it should be straightforward.

mcflugen · 2024-10-10T21:33:10Z

The fix looks like it will be straightforward but I'm not sure how to reproduce the problem so that we know we've fixed it. It looks like the current set of (Landlab) tests already test this case.

mcflugen · 2024-10-11T02:41:31Z

@mdpiper I hope that landlab/landlab#2015 fixes this issue. The error you were seeing was caused by the unravel cython function being passed an int32 array were it was expected an int64 array (on Windows long is, sometimes anyway, 32 bit). I've fixed that issue with unravel but I'm not certain why it was getting the int32 array in the first place.

Maybe Windows runners on GA are 32 bit?

mdpiper · 2024-10-11T17:09:55Z

🎉

mcflugen

👍

mdpiper and others added 19 commits October 2, 2024 23:48

Remove duplicate install of testing dependencies

f7472b4

Require latest version of model_metadata

748605a

Remove obsolete esmpy requirement

e6d10fa

Don't install components for testing

bfb47fc

The subsequent call to `mamba install` updates *shapely* to 2.0, which breaks the <1.9 requirement from #164.

import UnitSystem from gimli.units

3a0c43b

remove redundant gimli.units unit tests

ce9508f

checkout gimli.units raises UdunitsError

556183d

Consolidate requirements in one install step

4d00331

Fix typos

dcc0514

Roll back model_metadata for older models

55ea7d1

Remove shapely version cap

4209cb2

Convert shapely proxy object to geometries

93519d5

See https://shapely.readthedocs.io/en/stable/migration.html#other-deprecated-functionality

Remove scipy version cap

54db6ab

Mark fail when extrapolating with previous and next methods

ac77490

See scipy/scipy#16997 (comment)

Check for AssertionError instead of ValueError

feef80e

Revert "Don't install components for testing"

e68392b

This reverts commit bfb47fc. The shapely version issue is resolved.

Merge branch 'master' into mdpiper/fixes-for-shapely-and-scipy-updates

0de19e9

Limit numpy<2

9e8e331

See #173.

Set numpy print options to legacy=1.21

ad59ace

Without this setting, the value of legacy=1.25 was being inherited from Landlab, which isn't allowed in the installed version of numpy<2 (currently 1.26.4).

Add python=3.11 to tests

4d00e8d

mdpiper added 3 commits October 10, 2024 13:46

Switch from Mambaforge to Miniforge

18c4cde

Switch from mamba to conda

3958c95

Update versions of packaged actions

3b50477

mdpiper marked this pull request as ready for review October 10, 2024 20:28

mdpiper requested a review from mcflugen October 10, 2024 20:28

mcflugen mentioned this pull request Oct 11, 2024

Fix an issue with unraveling jaggedarray on Windows landlab/landlab#2015

Merged

7 tasks

mcflugen added 3 commits October 11, 2024 10:03

install requirements with pip

94faa80

turn off fail-fast

8c8664b

switch back to conda to install requirements

481f678

mcflugen approved these changes Oct 11, 2024

View reviewed changes

mdpiper merged commit df648b4 into master Oct 11, 2024
14 checks passed

mdpiper deleted the mdpiper/fixes-for-shapely-and-scipy-updates branch October 11, 2024 17:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for shapely and scipy updates #174

Fixes for shapely and scipy updates #174

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mcflugen commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mcflugen commented Oct 10, 2024

mcflugen commented Oct 10, 2024

mcflugen commented Oct 11, 2024

mdpiper commented Oct 11, 2024

mcflugen left a comment

Fixes for shapely and scipy updates #174

Fixes for shapely and scipy updates #174

Conversation

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mcflugen commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mdpiper commented Oct 10, 2024

mcflugen commented Oct 10, 2024

mcflugen commented Oct 10, 2024

mcflugen commented Oct 11, 2024

mdpiper commented Oct 11, 2024

mcflugen left a comment

Choose a reason for hiding this comment