Use flit & upgrade CI/CD workflows #391

CasperWA · 2022-10-11T12:57:39Z

Closes #389
Closes #390

Use `flit`

Combine all package information into pyproject.toml and use flit to build the package instead of setuptools.

Upgrade CI/CD workflows

Use SINTEF/ci-cd's callable workflows where applicable, since these will be updated automatically there and will minimize overhead in this project.

codecov · 2022-10-11T13:57:23Z

Codecov Report

Base: 93.13% // Head: 92.99% // Decreases project coverage by -0.15% ⚠️

Coverage data is based on head (7bdeaed) compared to base (ff4b31d).
Patch has no changes to coverable lines.

❗ Current head 7bdeaed differs from pull request most recent head a1eaee3. Consider uploading reports for the commit a1eaee3 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #391      +/-   ##
===========================================
- Coverage    93.13%   92.99%   -0.15%     
===========================================
  Files           33       31       -2     
  Lines         1370     1341      -29     
===========================================
- Hits          1276     1247      -29     
  Misses          94       94

Flag	Coverage Δ
aiida	`89.33% <ø> (-0.23%)`	⬇️
mongo	`79.41% <ø> (-0.44%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiida_optimade/cli/cmd_run.py	`69.69% <0.00%> (-0.90%)`	⬇️
aiida_optimade/cli/utils.py	`75.00% <0.00%> (ø)`
aiida_optimade/cli/cmd_calc.py	`83.87% <0.00%> (ø)`
aiida_optimade/cli/cmd_init.py	`99.20% <0.00%> (ø)`
aiida_optimade/cli/cmd_aiida_optimade.py	`100.00% <0.00%> (ø)`
aiida_optimade/utils.py
aiida_optimade/__init__.py

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Combine all package information into pyproject.toml and use `flit` to build the package instead of setuptools.

It's erroneously removing wrapping quotations.

…update-ci-cd-use-flit

CasperWA · 2022-11-29T14:30:03Z

@unkcpz I don't know what to do about this ClosedStorage exception that's now coming from AiiDA: https://github.com/aiidateam/aiida-optimade/actions/runs/3566504040/jobs/5993029685#step:7:15140

Is this something you can help with perhaps?

unkcpz · 2022-11-29T15:00:15Z

Not sure where it comes from. But where is this get_valid_id fixture used? There are aiida_profile fixture that can be used to have a temporary database.

CasperWA · 2022-11-30T08:33:56Z

Not sure where it comes from. But where is this get_valid_id fixture used? There are aiida_profile fixture that can be used to have a temporary database.

get_valid_id I suppose is for retrieving either a valid AiiDA Node ID or a valid OPTIMADE entry ID? I don't remember. It should hopefully be clear from the fixture itself?

unkcpz · 2022-12-07T18:53:31Z

Hi @CasperWA, sorry for the late reply.

I did some tests with directly using the fixture aiida_profile provided by aiida, but the same exception was for the test.
It is not just the get_valid_id that causes the issue, I think all query has the problem.
In this PR, do you only implement the type check or do you also implement new features such as multithreading or asynchronous query?
We had the issue in AiiDAlab that using threading caused the problem since now the Node queried is alive during the session, if there are other threading using the node, it cause the problem that the session is closed.

pinning @chrisjsewell for help, is this has the same root as the AiiDAlab one?

I attach the error traceback below from my local test, simply run pytest tests/server/routers/test_structures.py::test_get_next_responses -v -s.

Traceback

>       response: "Response" = client.get("/structures" + f"?page_limit={page_limit}")

tests/server/routers/test_structures.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/requests/sessions.py:600: in get
    return self.request("GET", url, **kwargs)
tests/server/utils.py:111: in request
    return super().request(
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/testclient.py:473: in request
    return super().request(
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/requests/sessions.py:587: in request
    resp = self.send(prep, **send_kwargs)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/requests/sessions.py:701: in send
    r = adapter.send(request, **kwargs)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/testclient.py:267: in send
    raise exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/testclient.py:264: in send
    portal.call(self.app, scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/anyio/from_thread.py:283: in call
    return cast(T_Retval, self.start_task_soon(func, *args).result())
../../../miniconda3/envs/optimade-dev/lib/python3.10/concurrent/futures/_base.py:458: in result
    return self.__get_result()
../../../miniconda3/envs/optimade-dev/lib/python3.10/concurrent/futures/_base.py:403: in __get_result
    raise self._exception
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/anyio/from_thread.py:219: in _call_func
    retval = await retval
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/fastapi/applications.py:270: in __call__
    await super().__call__(scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/applications.py:124: in __call__
    await self.middleware_stack(scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/errors.py:184: in __call__
    raise exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/errors.py:162: in __call__
    await self.app(scope, receive, _send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:72: in __call__
    response = await self.dispatch_func(request, call_next)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/optimade/server/middleware.py:433: in dispatch
    response = await call_next(request)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:46: in call_next
    raise app_exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:36: in coro
    await self.app(scope, request.receive, send_stream.send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:72: in __call__
    response = await self.dispatch_func(request, call_next)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/optimade/server/middleware.py:270: in dispatch
    response = await call_next(request)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:46: in call_next
    raise app_exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:36: in coro
    await self.app(scope, request.receive, send_stream.send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:72: in __call__
    response = await self.dispatch_func(request, call_next)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/optimade/server/middleware.py:109: in dispatch
    response = await call_next(request)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:46: in call_next
    raise app_exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:36: in coro
    await self.app(scope, request.receive, send_stream.send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:72: in __call__
    response = await self.dispatch_func(request, call_next)
aiida_optimade/middleware.py:38: in dispatch
    response = await call_next(request)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:46: in call_next
    raise app_exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:36: in coro
    await self.app(scope, request.receive, send_stream.send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:72: in __call__
    response = await self.dispatch_func(request, call_next)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/optimade/server/middleware.py:74: in dispatch
    response = await call_next(request)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:46: in call_next
    raise app_exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/base.py:36: in coro
    await self.app(scope, request.receive, send_stream.send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/cors.py:84: in __call__
    await self.app(scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/exceptions.py:75: in __call__
    raise exc
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/middleware/exceptions.py:64: in __call__
    await self.app(scope, receive, sender)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py:21: in __call__
    raise e
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py:18: in __call__
    await self.app(scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/routing.py:680: in __call__
    await route.handle(scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/routing.py:275: in handle
    await self.app(scope, receive, send)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/routing.py:65: in app
    response = await func(request)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/fastapi/routing.py:235: in app
    raw_response = await run_endpoint_function(
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/fastapi/routing.py:163: in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/starlette/concurrency.py:41: in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/anyio/to_thread.py:31: in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:937: in run_sync_in_worker_thread
    return await future
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:867: in run
    result = context.run(func, *args)
aiida_optimade/routers/utils.py:167: in wrapper
    value = func(*args, **kwargs)
aiida_optimade/routers/structures.py:43: in get_structures
    return get_entries(
aiida_optimade/routers/utils.py:85: in get_entries
    ) = collection.find(params)
aiida_optimade/entry_collections.py:211: in find
    results, more_data_available = self._run_db_query(
aiida_optimade/entry_collections.py:286: in _run_db_query
    for entity in self._find_all(**criteria):
aiida_optimade/entry_collections.py:335: in _find_all
    res = query.all()
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/orm/querybuilder.py:1088: in all
    matches = list(self.iterall(batch_size=batch_size))
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/orm/querybuilder.py:1050: in iterall
    for item in self._impl.iterall(self.as_dict(), batch_size):
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/storage/psql_dos/orm/querybuilder/main.py:165: in iterall
    with self.query_session(data) as build:
../../../miniconda3/envs/optimade-dev/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/storage/psql_dos/orm/querybuilder/main.py:233: in query_session
    query = self.get_query(data)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/storage/psql_dos/orm/querybuilder/main.py:225: in get_query
    self._query_cache = self._build(data)
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/storage/psql_dos/orm/querybuilder/main.py:277: in _build
    query = self.get_session().query()
../../../miniconda3/envs/optimade-dev/lib/python3.10/site-packages/aiida/storage/psql_dos/orm/querybuilder/main.py:142: in get_session
    return self._backend.get_session()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aiida.storage.psql_dos.backend.PsqlDosBackend object at 0x7fae3258f1f0>

    def get_session(self) -> Session:
        """Return an SQLAlchemy session bound to the current thread."""
        if self._session_factory is None:
>           raise ClosedStorage(str(self))
E           aiida.common.exceptions.ClosedStorage: Storage for 'test_profile' [closed] @ postgresql://aiida:***@localhost:43279/aiida_db / file:///tmp/tmpet8e91g4/test_repo

…update-ci-cd-use-flit

Both are sub-dependencies that should be fixed by pymatgen and aiida-core, respectively.

CasperWA added 8 commits October 21, 2022 13:21

Use flit

a496989

Combine all package information into pyproject.toml and use `flit` to build the package instead of setuptools.

Adjust YAML formatter hook

b513be0

Exclude dependabot YAML for formatter

a3aa7de

It's erroneously removing wrapping quotations.

Remove mentions of requirements and setup

9f1c1ba

Remove timeout times in CI

88742d1

Remove backend as a parameter in CI pytest

1fe91ee

Use SINTEF/ci-cd for standard workflows

91cec03

Temp: Use pylint

038c5fe

CasperWA force-pushed the cwa/close-389-390-update-ci-cd-use-flit branch from 2f1ecad to 038c5fe Compare October 21, 2022 11:23

CasperWA added 10 commits October 21, 2022 13:24

Update hook names for transparency

5b7eb5a

Use TYPE_CHECKING to avoid cyclic imports

8681335

Adhere to pylint in the whole code base

ad5ae48

Merge remote-tracking branch 'origin/develop' into cwa/close-389-390-…

9d646b5

…update-ci-cd-use-flit

Fix indentation

c61ff10

Return updated ci job python dependencies install

0e6c6aa

Handle if --filename is not given

ee95f2a

Move import to try and avoid cyclic imports

4df189f

Avoid covering TYPE_CHECKING block

404e53f

Initialize __node to None

9bed03a

CasperWA added 6 commits January 19, 2023 17:22

Merge remote-tracking branch 'origin/develop' into cwa/close-389-390-…

ffa706a

…update-ci-cd-use-flit

Run pylint & safety in CI

9c867ed

Ignore safety errors for mpmath & SQLALchemy

4161ff1

Both are sub-dependencies that should be fixed by pymatgen and aiida-core, respectively.

Reintroduce MANIFEST.in

ec94b5b

Re-remove MANIFEST.in - update pyproject instead

30240c8

Updates trying to fix tests

a1eaee3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use flit & upgrade CI/CD workflows #391

Use flit & upgrade CI/CD workflows #391

CasperWA commented Oct 11, 2022

codecov bot commented Oct 11, 2022 •

edited

Loading

CasperWA commented Nov 29, 2022

unkcpz commented Nov 29, 2022

CasperWA commented Nov 30, 2022

unkcpz commented Dec 7, 2022 •

edited

Loading

Use flit & upgrade CI/CD workflows #391

Are you sure you want to change the base?

Use flit & upgrade CI/CD workflows #391

Conversation

CasperWA commented Oct 11, 2022

Use flit

Upgrade CI/CD workflows

codecov bot commented Oct 11, 2022 • edited Loading

Codecov Report

CasperWA commented Nov 29, 2022

unkcpz commented Nov 29, 2022

CasperWA commented Nov 30, 2022

unkcpz commented Dec 7, 2022 • edited Loading

Use `flit`

codecov bot commented Oct 11, 2022 •

edited

Loading

unkcpz commented Dec 7, 2022 •

edited

Loading