Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve pylint benchmarks #178

Merged
merged 18 commits into from
Sep 1, 2020
Merged

Resolve pylint benchmarks #178

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
a740c7f
resolved pylint complaints in distribution benchmarks
pnkraemer Aug 26, 2020
0eea0d5
cleaned up pylint messages in linearsolver, too
pnkraemer Aug 26, 2020
e222448
pulled out random spd matrix
pnkraemer Aug 26, 2020
61289fd
finalised distribution and linearsolver
pnkraemer Aug 26, 2020
3138802
blacked results
pnkraemer Aug 26, 2020
145f311
black messed up my pylint ignore
pnkraemer Aug 26, 2020
6f1e9aa
removed benchmark directory from tox pylint ignore
pnkraemer Aug 26, 2020
37c15a3
blacked
pnkraemer Aug 26, 2020
329f1f7
Merge branch 'master' into resolve_pylint_benchmarks
pnkraemer Aug 27, 2020
821b1aa
merge master
pnkraemer Aug 31, 2020
59a0c03
Merge branch 'resolve_pylint_benchmarks' of https://github.com/pnkrae…
pnkraemer Aug 31, 2020
1d35736
re-fix ivpsolver and linearsolver
pnkraemer Aug 31, 2020
6376137
added pylint ignores, renamed property into method
pnkraemer Aug 31, 2020
84c2f57
renamed random spd matrix
pnkraemer Aug 31, 2020
1ed0a9c
globally ignored missing-function-docstring
pnkraemer Aug 31, 2020
00a6894
documented benchmark change in tox file
pnkraemer Aug 31, 2020
0c37ca5
blacked
pnkraemer Aug 31, 2020
7269727
mini docstring updates
pnkraemer Aug 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions benchmarks/benchmark_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
"""Reocurring constants and methods for benchmarking functions."""

import numpy as np


RANDOM_5x5_SPD_MATRIX = np.array(
[
[2.3, -2.3, 3.5, 4.2, 1.8],
[-2.3, 3.0, -3.5, -4.8, -1.9],
[3.5, -3.5, 6.9, 5.8, 0.8],
[4.2, -4.8, 5.8, 10.1, 6.3],
[1.8, -1.9, 0.8, 6.3, 12.1],
]
)
52 changes: 26 additions & 26 deletions benchmarks/distribution.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@
import probnum.prob as prob
import probnum.linalg.linops as linops

from benchmarks.benchmark_utils import RANDOM_5x5_SPD_MATRIX

# Module level variables
distribution_names = [
DISTRIBUTION_NAMES = [
"univar_normal",
"multivar_normal",
"matrixvar_normal",
Expand All @@ -22,15 +24,7 @@ def get_randvar(distribution_name):
Return a random variable for a given distribution name
"""
# Distribution Means and Covariances
spd_mat = np.array(
[
[2.3, -2.3, 3.5, 4.2, 1.8],
[-2.3, 3.0, -3.5, -4.8, -1.9],
[3.5, -3.5, 6.9, 5.8, 0.8],
[4.2, -4.8, 5.8, 10.1, 6.3],
[1.8, -1.9, 0.8, 6.3, 12.1],
]
)
spd_mat = RANDOM_5x5_SPD_MATRIX
mean_0d = np.random.rand()
mean_1d = np.random.rand(5)
mean_2d_mat = spd_mat
Expand Down Expand Up @@ -60,25 +54,28 @@ class Functions:
Benchmark various functions of distributions.
"""

param_names = ["dist", "property"]
params = [distribution_names, ["pdf", "logpdf", "cdf", "logcdf"]]
param_names = ["dist", "method"]
params = [DISTRIBUTION_NAMES, ["pdf", "logpdf", "cdf", "logcdf"]]

def setup(self, dist, property):
def setup(self, dist, method):
# pylint: disable=missing-function-docstring,attribute-defined-outside-init
# pylint: disable=unused-argument
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

# pylint: disable=unused-argument
# pylint: disable=missing-function-docstring

will appear in almost every benchmark and should if possible be ignored for all benchmarks.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring warning makes sense to be ignored globally IMO (the docstrings seem to be super repetitive anyway: "Time method xyz", "Benchmark memory usage of xyz". The unused-argument I am more hesitant about...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put the ignore for unused-argument back into the tox file. Any idea where we can best document that this is not one of the "we have not resolved this yet" ignores but a "final" ignore? I put a comment into the tox ini file, but if you have a better idea please share.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe @nathanaelbosch knows a way to ignore specific warnings in a certain folder, however judging from StackOverflow one can only achieve this by having a .pylintrc file in benchmarks and then having two pylint statements (which we already have anyway).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marvinpfoertner also looked into it and, if I'm not mistaken, came to the conclusion that it's unfortunately not possible to have configs on a per-directory basis.

In general I'm definitely open for ideas to move these exceptions out of the tox.ini, since the current way does not feel ideal.

self.randvar = get_randvar(distribution_name=dist)
self.eval_point = np.random.uniform(self.randvar.shape)
self.quantile = np.random.uniform(self.randvar.shape)
self.eval_point = np.random.uniform(size=self.randvar.shape)
self.quantile = np.random.uniform(size=self.randvar.shape)

def time_distr_functions(self, dist, property):
"""Times evaluation of the pdf, logpdf, cdf and logcdf."""
def time_distr_functions(self, dist, method):
"""Time evaluation of the pdf, logpdf, cdf and logcdf."""
# pylint: disable=unused-argument
try:
if property == "pdf":
self.randvar.distribution.pdf(x=self.eval_point)
elif property == "logpdf":
if method == "pdf":
self.randvar.distribution.pdf(x=self.eval_point)
elif property == "cdf":
self.randvar.distribution.pdf(x=self.quantile)
elif property == "logcdf":
self.randvar.distribution.pdf(x=self.quantile)
elif method == "logpdf":
self.randvar.distribution.logpdf(x=self.eval_point)
elif method == "cdf":
self.randvar.distribution.cdf(x=self.quantile)
elif method == "logcdf":
self.randvar.distribution.logcdf(x=self.quantile)
except NotImplementedError:
pass

Expand All @@ -89,17 +86,20 @@ class Sampling:
"""

param_names = ["dist"]
params = [distribution_names]
params = [DISTRIBUTION_NAMES]

def setup(self, dist):
# pylint: disable=missing-function-docstring,attribute-defined-outside-init
np.random.seed(42)
self.n_samples = 1000
self.randvar = get_randvar(distribution_name=dist)

def time_sample(self, dist):
"""Times sampling from this distribution."""
"""Time sampling from this distribution."""
# pylint: disable=unused-argument
self.randvar.sample(self.n_samples)

def peakmem_sample(self, dist):
"""Peak memory of sampling process."""
# pylint: disable=unused-argument
self.randvar.sample(self.n_samples)
50 changes: 29 additions & 21 deletions benchmarks/linearsolvers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import scipy.sparse

from probnum.linalg import problinsolve
from benchmarks.benchmark_utils import RANDOM_5x5_SPD_MATRIX


def load_poisson_linear_system():
Expand All @@ -21,69 +22,76 @@ def load_poisson_linear_system():

Linear system resulting from discretization on an elliptic grid.
"""
# pylint: disable=invalid-name
fpath = os.path.join(os.path.dirname(__file__), "../tests/resources")
A = scipy.sparse.load_npz(file=fpath + "/matrix_poisson.npz")
f = np.load(file=fpath + "/rhs_poisson.npy")
return A, f


class LinSolve:
"""
Benchmark solving a linear system.
"""
"""Benchmark solving a linear system."""

param_names = ["system"]
params = [["sparse", "dense"]] # , "large-scale"]

def setup(self, system):
# pylint: disable=missing-function-docstring,attribute-defined-outside-init

# Seed
np.random.seed(42)

if system == "sparse":
self.A, self.b = load_poisson_linear_system()
(
self.A,
self.b,
) = load_poisson_linear_system() # pylint: disable=invalid-name
elif system == "dense":
self.A = np.array(
[
[2.3, -2.3, 3.5, 4.2, 1.8],
[-2.3, 3.0, -3.5, -4.8, -1.9],
[3.5, -3.5, 6.9, 5.8, 0.8],
[4.2, -4.8, 5.8, 10.1, 6.3],
[1.8, -1.9, 0.8, 6.3, 12.1],
]
)
self.A = RANDOM_5x5_SPD_MATRIX
self.b = np.random.normal(size=self.A.shape[0])
elif system == "large-scale":
self.A = None
self.b = None

def time_solve(self, system):
# Solve linear system
self.xhat, self.Ahat, self.Ainvhat, _ = problinsolve(A=self.A, b=self.b)
"""Time solving a linear system"""
# pylint: disable=unused-argument
problinsolve(A=self.A, b=self.b)

def mem_solve(self, system):
# Solve linear system
self.xhat, self.Ahat, self.Ainvhat, _ = problinsolve(A=self.A, b=self.b)
"""Time solving a linear system"""
# pylint: disable=unused-argument
problinsolve(A=self.A, b=self.b)

def peakmem_solve(self, system):
# Solve linear system
self.xhat, self.Ahat, self.Ainvhat, _ = problinsolve(A=self.A, b=self.b)
"""Time solving a linear system"""
# pylint: disable=unused-argument
problinsolve(A=self.A, b=self.b)


class PosteriorDist:
"""Benchmark sampling from the posterior distribution."""

param_names = ["output"]
params = [["solution", "matrix", "matrix_inverse"]]

def setup(self, output):
# pylint: disable=missing-function-docstring,attribute-defined-outside-init
# pylint: disable=unused-argument

# Sparse system
self.A, self.b = load_poisson_linear_system()
self.A, self.b = load_poisson_linear_system() # pylint: disable=invalid-name

# Solve linear system
self.xhat, self.Ahat, self.Ainvhat, _ = problinsolve(A=self.A, b=self.b)
self.xhat, self.Ahat, self.Ainvhat, _ = problinsolve(
A=self.A, b=self.b
) # pylint: disable=invalid-name

# Benchmark parameters
self.n_samples = 10

def time_sample(self, output):
"""Time sampling from the posterior distribution"""
if output == "solution":
self.xhat.sample(self.n_samples)
elif output == "matrix":
Expand Down
1 change: 0 additions & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,3 @@ commands =
pylint src/probnum/quad --disable="attribute-defined-outside-init,too-few-public-methods,redefined-builtin,arguments-differ,unused-argument,missing-module-docstring"
pylint src/probnum/utils --disable="duplicate-code,missing-module-docstring,missing-function-docstring"
pylint tests --disable="line-too-long,duplicate-code,missing-class-docstring,unnecessary-pass,unused-variable,protected-access,attribute-defined-outside-init,no-self-use,abstract-class-instantiated,too-many-arguments,too-many-instance-attributes,too-many-locals,unused-argument,fixme,missing-module-docstring,missing-function-docstring"
pylint benchmarks --disable="attribute-defined-outside-init,unused-argument,redefined-builtin,missing-class-docstring,duplicate-code,missing-function-docstring"