Line Search for Newton-Raphson #194

yash2798 · 2023-07-11T23:43:17Z

No description provided.

codecov · 2023-07-11T23:49:37Z

Codecov Report

Merging #194 (f8b6685) into master (81e9164) will decrease coverage by 1.93%.
The diff coverage is 67.92%.

❗ Current head f8b6685 differs from pull request most recent head 137d8fd. Consider uploading reports for the commit 137d8fd to get more accurate results

@@            Coverage Diff             @@
##           master     #194      +/-   ##
==========================================
- Coverage   94.42%   92.49%   -1.93%     
==========================================
  Files           7        7              
  Lines         699      746      +47     
==========================================
+ Hits          660      690      +30     
- Misses         39       56      +17

Files Changed	Coverage Δ
src/NonlinearSolve.jl	`100.00% <ø> (ø)`
src/jacobian.jl	`82.40% <0.00%> (-8.41%)`	⬇️
src/utils.jl	`81.42% <ø> (ø)`
src/raphson.jl	`91.50% <83.72%> (-5.60%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

added objective func value output

simple jacobian calculation

1. step variable du1 for out of place functions. 2. added linesearch func

1. export LineSearches Alternative: export LineSearches: ... ?

1. modified cache 2. final tweaks

oscardssmith · 2023-07-20T14:19:01Z

src/jacobian.jl

@@ -149,3 +149,19 @@ function jacobian_autodiff(f, x::AbstractArray, nonlinfun, alg)
            jac_prototype = jac_prototype, chunksize = chunk_size),
        num_of_chunks)
 end
+
+function simple_jacobian(cache, x::Number)


shouldn't the nonlinear solve be responsible for providing the jacobian? What if the problem doesn't support autodiff?

So should I just use FiniteDiff when autodiff is passed as false ? That should be a correct approach?

This is already handled by other dispatches. Why does this exist?

yes, however in the current dispatch, the jacobian is calculated at cache.u, i.e. the dispatch takes in cache as an argument and not the x. But in when linesearch is performed, we need the jacobian at multiple x's (for multiple step lengths $\alpha_{k}$), so we can't just do jacobian(f, cache) as in the current dispatch.

for example like here (https://github.com/JuliaNLSolvers/LineSearches.jl/blob/ded667a80f47886c77d67e8890f6adb127679ab4/src/strongwolfe.jl#L84)

i can't think of any other way to approach this, any inputs will be helpful?

although i have an idea like this roughly

function simple_jacobian(cache, x)
cache.tmp_var = cache.u
cache.u = x
J = jacobian(cache, f) ## our current dispatch
cache.u = cache.tmp_var
end

this way we don't have to deal with sparsity and AD/Finite diff separately

Yes, it should be something like that reusing the existing machinery. But the vjp shouldn't build the Jacobian at all: you don't actually need to calculate the Jacobian for most of this.

yes you are absolutely right. so we need $J^{T}v = (v^{T}J)^{T}$. We can calculate vjp $v^{T}J$. are there any routines in SparseDiffTools.jl or do we need to do this kind of 'in-situ'

if in-situ, can you point me to something that i can take help from regarding reverse-mode AD, i am not very well versed with it.

Yes, that's precisely what Reverse-Mode AD calculates without calculating the Jacobian. This is why I want @avik-pal to take a look and see how that would mesh with #206

src/raphson.jl

yash2798 · 2023-07-20T15:00:37Z

src/jacobian.jl

@@ -149,3 +149,19 @@ function jacobian_autodiff(f, x::AbstractArray, nonlinfun, alg)
            jac_prototype = jac_prototype, chunksize = chunk_size),
        num_of_chunks)
 end
+
+function simple_jacobian(cache, x::Number)


So should I just use FiniteDiff when autodiff is passed as false ? That should be a correct approach?

ChrisRackauckas · 2023-07-24T14:24:44Z

So should I just use FiniteDiff when autodiff is passed as false ? That should be a correct approach?

Yes, an internal Jacobian function is a bit suspect. Definitely use FiniteDiff+ForwardDiff (SparseDiffTools) when possible.

src/utils.jl

yash2798 · 2023-09-17T00:47:34Z

@ChrisRackauckas i did some preliminary tests and they seem to perform well (similar results as NLsolve) but I have concerns regarding the efficiency (or "code performance") in general which I am highlighting.

yash2798 · 2023-09-17T00:49:56Z

src/raphson.jl

@@ -172,6 +182,43 @@ function perform_step!(cache::NewtonRaphsonCache{false})
    return nothing
 end

+function objective_linesearch(cache::NewtonRaphsonCache) ## returns the objective functions required in LS


This function objective_linesearch returns 3 functions and is called in a loop (so it might get called multiple times). is this an efficient approach?

yash2798 · 2023-09-17T00:51:23Z

src/raphson.jl

+    ϕ(α) = fo(u .- α .* du1)
+
+    function dϕ(α)
+        g!(g_o, u .- α .* du1)
+        return dot(g_o, -du1)
+    end
+
+    function ϕdϕ(α)
+        return (fg!(g_o, u .- α .* du1), dot(g_o, -du1))
+    end


Similarly, these 3 functions are defined each time the perform_linesearch! function is called. Can this lead to performance issues?

ChrisRackauckas · 2023-09-17T03:33:12Z

src/raphson.jl

+    end
+
+    function g!(g_o, x)
+        mul!(g_o, simple_jacobian(cache, x)', value_f(cache, x)) 


This is not a good way to compute this.

This should be a vjp with reversemode ad?

ChrisRackauckas · 2023-09-17T03:33:20Z

src/raphson.jl

+    @unpack f, p = cache
+
+    function fo(x)
+        return dot(value_f(cache, x), value_f(cache, x)) / 2


Why is the same function called twice?

ChrisRackauckas · 2023-09-17T04:33:38Z

@avik-pal I assume you'll need to incorporate this into #206

avik-pal · 2023-09-20T21:45:51Z

Can be closed in favor of #203.

avik-pal · 2023-09-20T21:53:08Z

I incorporated the line search in that PR. It doesn't construct Jacobians and instead uses VJPs and also caches the function construction during the nonlinear solver cache construction.

started linesearch implementation

866fa81

yash-rs added 16 commits July 13, 2023 02:47

added objective functions

1983842

modified objective and added helper func

c54601f

reversing changes

879e118

modified cache and objective func final dispatch

58774f1

Update utils.jl

4b04e2a

added objective func value output

Update jacobian.jl

72eb6b8

simple jacobian calculation

objective correction

95c89a4

quick fixes

929552d

improve simple_jacobian

f6920f7

added linesearch functions

4982846

Update raphson.jl

b5d9f3e

1. step variable du1 for out of place functions. 2. added linesearch func

export

e1a2da9

1. export LineSearches Alternative: export LineSearches: ... ?

modified jac calc

2593f9a

removed test exports

b604e5b

get_iip improvement

1591dff

Update raphson.jl

c74c78d

1. modified cache 2. final tweaks

yash2798 changed the title ~~[WIP] Line Search for Newton-Raphson~~ Line Search for Newton-Raphson Jul 20, 2023

yash2798 marked this pull request as ready for review July 20, 2023 00:47

yash-rs added 2 commits July 20, 2023 14:24

Merge branch 'master' into ys/linesearch

ea1b466

--

ad4ea57

oscardssmith reviewed Jul 20, 2023

View reviewed changes

src/raphson.jl Show resolved Hide resolved

oscardssmith reviewed Jul 20, 2023

View reviewed changes

src/raphson.jl Show resolved Hide resolved

yash2798 commented Jul 24, 2023

View reviewed changes

yash2798 marked this pull request as draft July 25, 2023 01:13

yash2798 changed the title ~~Line Search for Newton-Raphson~~ [WIP] Line Search for Newton-Raphson Jul 25, 2023

avik-pal reviewed Aug 18, 2023

View reviewed changes

src/utils.jl Outdated Show resolved Hide resolved

yash-rs added 6 commits August 30, 2023 00:33

Merge branch 'master' into ys/linesearch

18c19ad

Merge branch 'master' into ys/linesearch

0871967

isinplace use

8752556

isinplace use

ff524b0

minor sign fix

7673c85

Merge branch 'master' into ys/linesearch

f8b6685

yash2798 commented Sep 17, 2023

View reviewed changes

more minor fixes

137d8fd

yash2798 changed the title ~~[WIP] Line Search for Newton-Raphson~~ Line Search for Newton-Raphson Sep 17, 2023

yash2798 marked this pull request as ready for review September 17, 2023 01:22

ChrisRackauckas reviewed Sep 17, 2023

View reviewed changes

ChrisRackauckas closed this Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Line Search for Newton-Raphson #194

Line Search for Newton-Raphson #194

yash2798 commented Jul 11, 2023

codecov bot commented Jul 11, 2023 •

edited

Loading

oscardssmith Jul 20, 2023

yash2798 Jul 20, 2023

ChrisRackauckas Sep 17, 2023

yash2798 Sep 17, 2023

yash2798 Sep 17, 2023 •

edited

Loading

yash2798 Sep 17, 2023

ChrisRackauckas Sep 17, 2023

yash2798 Sep 17, 2023

yash2798 Sep 18, 2023

ChrisRackauckas Sep 18, 2023

yash2798 Jul 20, 2023

ChrisRackauckas commented Jul 24, 2023

yash2798 commented Sep 17, 2023 •

edited

Loading

yash2798 Sep 17, 2023

yash2798 Sep 17, 2023

ChrisRackauckas Sep 17, 2023

ChrisRackauckas Sep 17, 2023

ChrisRackauckas Sep 17, 2023

ChrisRackauckas commented Sep 17, 2023

avik-pal commented Sep 20, 2023

avik-pal commented Sep 20, 2023

Line Search for Newton-Raphson #194

Line Search for Newton-Raphson #194

Conversation

yash2798 commented Jul 11, 2023

codecov bot commented Jul 11, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yash2798 Sep 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisRackauckas commented Jul 24, 2023

yash2798 commented Sep 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisRackauckas commented Sep 17, 2023

avik-pal commented Sep 20, 2023

avik-pal commented Sep 20, 2023

codecov bot commented Jul 11, 2023 •

edited

Loading

yash2798 Sep 17, 2023 •

edited

Loading

yash2798 commented Sep 17, 2023 •

edited

Loading