Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem with inverse projection of "ortho" #1729

Closed
dankelley opened this issue Jul 15, 2021 · 22 comments
Closed

problem with inverse projection of "ortho" #1729

dankelley opened this issue Jul 15, 2021 · 22 comments

Comments

@dankelley
Copy link
Contributor

I may have encountered an sf problem with the ortho projection, in cases with a viewpoint that is not on the equatorial plane. I am on macOS Monterey (Version 12.0 beta (21A5268h).

The reprex pasted below illustrates. The points are for a lon-lat grid transformed to x-y space using the ortho projection. Red dots are used if the point can be converted back from x-y to lon-lat, and blue otherwise. Thus, blue indicates a problem, and I see this in thin patches along the top "shoulders" of the view, and in larger patches along the sides of the bottom region.

This problem shows up on my macOS system in three versions of sf:

  1. The 1.0.0 binary from CRAN
  2. The 1.0.1 tarball, built from the source provided on CRAN
  3. the present development version, built following the directions in the README file

A colleague using an earlier version of sf revealed that it was ok in that system. I've also found that I get all red dots if use rgdal::project() for the computations.

From these tests, it seems to me that there is something worth exploring. I hope my test is simple enough to be useful. Of course, I would be very happy to do further tests, if requested.

I hope this comment is of some use.

PS. more discussion of why I'm doing this is at dankelley/oce#1848, although it's not fully-explained, the issue being more in the form of a diary of an idea I'm trying for the oce package.

# Demonstrate problem with +proj=ortho forward/backward
library(sf)
#> Linking to GEOS 3.9.1, GDAL 3.3.1, PROJ 7.2.1
sessionInfo()
#> R version 4.1.0 (2021-05-18)
#> Platform: x86_64-apple-darwin17.0 (64-bit)
#> Running under: macOS Big Sur 10.16
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib
#> 
#> locale:
#> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] sf_1.0-2
#> 
#> loaded via a namespace (and not attached):
#>  [1] Rcpp_1.0.7         pillar_1.6.1       compiler_4.1.0     highr_0.9         
#>  [5] class_7.3-19       tools_4.1.0        digest_0.6.27      evaluate_0.14     
#>  [9] lifecycle_1.0.0    tibble_3.1.2       pkgconfig_2.0.3    rlang_0.4.11      
#> [13] reprex_2.0.0       DBI_1.1.1          yaml_2.2.1         xfun_0.24         
#> [17] e1071_1.7-7        withr_2.4.2        styler_1.5.1       stringr_1.4.0     
#> [21] dplyr_1.0.7        knitr_1.33         generics_0.1.0     fs_1.5.0          
#> [25] vctrs_0.3.8        tidyselect_1.1.1   classInt_0.4-3     grid_4.1.0        
#> [29] glue_1.4.2         R6_2.5.0           fansi_0.5.0        rmarkdown_2.9     
#> [33] purrr_0.3.4        magrittr_2.0.1     backports_1.2.1    ellipsis_0.3.2    
#> [37] htmltools_0.5.1.1  units_0.7-2        assertthat_0.2.1   utf8_1.2.1        
#> [41] KernSmooth_2.23-20 stringi_1.7.2      proxy_0.4-26       crayon_1.4.1
proj <- "+proj=ortho +lat_0=-20 +datum=WGS84 +no_defs"
proj0 <- "+proj=longlat +datum=WGS84 +no_defs"

grid <- expand.grid(lon=seq(-180,180,2.5), lat=seq(-90,90,2.5))
lonlat <- cbind(grid$lon, grid$lat)
xy <- sf_project(proj0, proj, lonlat, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
LONLAT <- sf_project(proj, proj0, xy, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
ok <- is.finite(LONLAT[,1]) & is.finite(LONLAT[,2])

par(mar=c(3,3,1,1), mgp=c(2,0.7,0))
plot(xy[,1]/1e3, xy[,2]/1e3, asp=1,
     xlab="Easting Coordinate [km]", ylab="Northing Coordinate [km]",
     pch=20, cex=0.3, col=ifelse(ok, "red", "blue"))
grid()

Created on 2021-07-15 by the reprex package (v2.0.0)

@dankelley
Copy link
Contributor Author

Update: a colleague using a windows machine and an earlier sf found that things worked (see the all-red dots at dankelley/oce#1848 (comment)).

@edzer
Copy link
Member

edzer commented Jul 15, 2021

It would be useful to have the output of sf::sf_extSoftVersion() on machines where things work, and where they don't.

@dankelley
Copy link
Contributor Author

Thanks.

This is on my version (which fails). I'll ask my colleague, on which it worked, to do the same, and I'll report back on those results (likely not until tomorrow).

> sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H
       "3.9.1"        "3.3.1"        "7.2.1"         "true"         "true"

@dankelley
Copy link
Contributor Author

dankelley commented Jul 16, 2021

Here are test results with
https://github.com/dankelley/oce/blob/EOW/sandbox/dk/eow/eow05.R; for diagrams and details, see dankelley/oce#1848

macOS R-4.1.0 sf-0.9.5 (@dankelley)

  • x and y span +-6400km, as expected
  • blue dots indicate 4 problematic regions: in thin patches near 11 and 13
    o'clock, with larger patches near 5 and 8 o'clock.
  • sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
       "3.9.1"        "3.3.1"        "7.2.1"         "true"         "true" 

macOS R-4.1.0 sf-1.0.0 binary from CRAN (@dankelley)

  • x and y span +-6400km, as expected
  • blue dots indicate 4 problematic regions: in thin patches near 11 and 13
    o'clock, and larger patches near 5 and 8 o'clock.
  • sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H           PROJ
       "3.8.1"        "3.2.1"        "7.2.1"         "true"         "true"        "7.2.1"

macOS R-4.1.0 sf-1.0.1, built locally from CRAN tarball (@dankelley)

  • x and y span +-6400km, as expected
  • blue dots indicate 4 problematic regions: in thin patches near 11 and 13
    o'clock, and larger patches near 5 and 8 o'clock.
  • sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H           PROJ
       "3.9.1"        "3.3.1"        "7.2.1"         "true"         "true"        "7.2.1"

windows-10-x64 R-4.0.2 sf-0.9.5 (@clayton33)

  • x and y span +-6400km, as expected
  • all red dots, indicating no uninvertible points on globe
  • sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
       "3.9.1"        "3.3.1"        "7.2.1"         "true"         "true" 

ubuntu-18.04.5 R-4.1.0 sf-0.9.5 (@richardsc)

  • markedly different diagram from other results
  • x spans only approx +- 2200km, but y spans +-6400km.
  • all red dots, but not covering full globe so perhaps even the forward
    transform is problematic
  • sf::sf_extSoftVersion()
          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H
       "3.8.0"        "3.0.4"        "7.0.0"         "true"         "true"

@paleolimbot
Copy link
Contributor

paleolimbot commented Jul 16, 2021

The problem is almost 100% the "new" ellipsoidal formulation of the orthographic projection. You can force the spherical version by using +R=6378137 +f=0. See https://proj.org/operations/projections/ortho.html#id1 .

library(sf)
#> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1

proj <- "+proj=ortho +lat_0=-20 +R=6378137 +f=0"
proj0 <- "OGC:CRS84"

grid <- expand.grid(lon=seq(-180,180,2.5), lat=seq(-90,90,2.5))
lonlat <- cbind(grid$lon, grid$lat)
xy <- sf_project(proj0, proj, lonlat, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
LONLAT <- sf_project(proj, proj0, xy, keep=TRUE)

ok <- is.finite(LONLAT[,1]) & is.finite(LONLAT[,2])

par(mar=c(3,3,1,1), mgp=c(2,0.7,0))
plot(xy[,1]/1e3, xy[,2]/1e3, asp=1,
     xlab="Easting Coordinate [km]", ylab="Northing Coordinate [km]",
     pch=20, cex=0.3, col=ifelse(ok, "red", "blue"))
grid()

Created on 2021-07-16 by the reprex package (v0.3.0)

@dankelley
Copy link
Contributor Author

As usual, Dewey is very smart, and very helpful!! I'm thinking that I'll be able to get a version to work without asking users to update a whole lot of things, just by adding a converter function, along the lines shown below. (I guess I'll want to add a test on the sf (etc) version number, to make it clear that this is a temporary workaround.)

Oh, and thanks, @paleolimbot, for pointing out that neat trick of using OGC:CRS84, which I'm assuming is better than the spelt-out version I was using. I've a lot to learn!

projFix <- function(CRS) # FIXME: check sf version (?)
{
    if (grepl("=ortho", CRS)) {
        if (!grepl("+R=", CRS))
            CRS <- paste(CRS, "+R=6378137")
        if (!grepl("+f=", CRS))
            CRS <- paste(CRS, "+f=0")
    }
    CRS
}

proj <- projFix("+proj=ortho +lat_0=-20 +datum=WGS84 +no_defs")
proj0 <- projFix("OGC:CRS84") # ? "+proj=longlat +datum=WGS84 +no_defs"

@paleolimbot
Copy link
Contributor

It's equivalent to +proj=longlat...it got left in there because I was wondering if there was an axis ordering problem.

Thee thing I did is a workaround...this is a great catch of a bug in PROJ and should get reported there!

@rsbivand
Copy link
Member

rsbivand commented Jul 17, 2021

Not a bug, a design decision to permit both spherical and ellipsoidal: OSGeo/PROJ@2e104e0. See also issue OSGeo/PROJ#397 and the mailing list thread from https://lists.osgeo.org/pipermail/proj/2016-July/007020.html, all based on http://www.hydrometronics.com/downloads/Ellipsoidal%20Orthographic%20Projection.pdf. The note on https://proj.org/operations/projections/ortho.html giving the work-around to retain legacy output is in the commit at OSGeo/PROJ@2e104e0. The PROJ 7.2.0 release states in Updates in NEWS:

Added ellipsoidal formulation of +proj=ortho (#2361)

PROJ does make breaking changes requiring downstream intervention to retain output consistency with earlier versions; this also happens when the values in proj.db are updated, not only when projections are revised. This is a good example of differences in perception of what output is desired.

@paleolimbot
Copy link
Contributor

paleolimbot commented Jul 17, 2021

I think regardless of ellipsoidal vs spherical, the output that @dankelley kindly provided is not intentional...the set of invert-able points should be roughly the same whether the earth is ellipsoidal or spherical, no?

@rsbivand
Copy link
Member

rsbivand commented Jul 17, 2021

I think that to say this is a bug, an app-level reprex using first projinfo to expose the transformation pipelines for both cases, then proj for both cases for a small number of points would be needed, and would have to establish where the problem is coming from. Yes, the behaviour is unexpected, but why internally isn't clear. When +lat=0, both look OK, don't they? So with spherical (now +R= +f=0) +lat= doesn't have to be 0, but with ellipsoid ortho, it looks as though +lat=0 is required? Also, because the grid includes everything (all 2.5 degree grid intersections, all the non-visible ones are returned Inf, aren't they?

> system("echo -180 -69.9 | proj +proj=ortho +lat_0=-20 +datum=WGS84")
*	*
> system("echo -180 -70 | proj +proj=ortho +lat_0=-20 +datum=WGS84")
-0.00	-6372985.71
> system("echo -180 90 | proj +proj=ortho +lat_0=0 +datum=WGS84")
-0.00	6356752.31
> system("echo -180 89 | proj +proj=ortho +lat_0=0 +datum=WGS84")
*	*
> system("echo -180 -69.9 | proj +proj=ortho +lat_0=-20 +R=6378137 +f=0")
*	*
> system("echo -180 -70 | proj +proj=ortho +lat_0=-20 +R=6378137 +f=0")
-0.00	-6378137.00
> system("echo -180 90 | proj +proj=ortho +lat_0=0 +R=6378137 +f=0")
-0.00	6378137.00
> system("echo -180 89 | proj +proj=ortho +lat_0=0 +R=6378137 +f=0")
*	*

Further, the problems are only for the inverse, as forward works as described, with no visible blue points if ok is constructed from xy, even with +lat=-50. One oddish reprex possibility:

> system("echo -180 -70 | proj +proj=ortho +lat_0=-20 +datum=WGS84")
-0.00	-6372985.71
> system("echo -0.00 -6372985.71 | proj -I +proj=ortho +lat_0=-20 +datum=WGS84")
*	*
> system("echo -180 -70 | proj +proj=ortho +lat_0=-20 +R=6378137 +f=0")
-0.00	-6378137.00
> system("echo -0.00 -6372985.71 | proj -I +proj=ortho +lat_0=-20 +R=6378137 +f=0")
180dW	72d18'10.495"S

Maybe +proj=ortho doesn't handle inverse projection well near its bounds?

Can anyone see how to use an environment variable to set the PJ_LOG_LEVEL to get the trace? PROJ_DEBUG=3 isn't helpful and doesn't pick up the traces.

@paleolimbot
Copy link
Contributor

It looks like the polar and equatorial versions are special-cased and that anything else has a convergence algorithm that starts at the centre point and quits after 20 iterations.

https://github.com/OSGeo/PROJ/blob/master/src/projections/ortho.cpp#L233-L274

I wonder if 20 iterations is too few for a reasonable approximation?

@rsbivand
Copy link
Member

@dankelley Is the whole globe inverse ellipsoidal ortho really essential for +lat_0= not special cased, given that the code as @paleolimbot indicates tries to iterate but gives up quite easily? It looks as though the unlinked EPSG guidance considered this use unlikely, which would explain the lack of effort in resolving it? Forward projection works, the problem is ellipsoidal inverse when +lat= is not special cased, so +lat_0=0 and a number of others do work but +lat_0=-20 ellipsoidal does not? How often is the global ellipsoidal inverse really needed?

@dankelley
Copy link
Contributor Author

@rsbivand thanks for the question, and your comments on this issue. For my own case, looking mainly at world views, the difference between spherical and ellipsoidal models is not critical, and I can stick with the spherical view, for which it seems there is no problem.

I cannot speak for others, of course, since it's difficult to predict what people will need in their work.

Those who really need an inverse in the ellipsoidal case could always set up their own minimization code to find it, starting with the spherical inverse as a guess. When I've done that in the past, I've found simple schemes (e.g. Nelder-Meade) work reasonably well. Still, if I really needed the ellipsoidal inverse at the moment, I'd start by increasing the iteration limit that @paleolimbot pointed out. Admittedly, I've not examined
https://github.com/OSGeo/PROJ/blob/23636dede21c8c47e660dcd880c91bff9304d4ca/src/projections/ortho.cpp#L261 and the line after it, to try to understand the pattern of refinement between iterations.

@rsbivand
Copy link
Member

So on balance should we raise an issue/create a PR to increase the limit? Or work around the difficulty by changing code to ensure that the legacy spherical code is used, which I think is the intention of the note in the documentation?

@paleolimbot
Copy link
Contributor

I played with this a bit and can't find anything in ortho.cpp that is off...the inverse seems to work and seems to be unchanged since the implementation. You can recreate my adventures here: https://gist.github.com/paleolimbot/6cecbac456a21030b9c2aaf5214e5e82 .

@rsbivand
Copy link
Member

Very interesting! See also:

library(rgdal)
list_coordOps(paste0(proj, " type=crs"), paste0(proj0, " type=crs"))
xy_rgdal <- project(lonlat, proj)
LONLAT_rgdal  <- project(lonlat, proj, inv=TRUE)
ok3 <- is.finite(LONLAT_rgdal[,1]) & is.finite(LONLAT_rgdal[,2])
plot(xy[,1]/1e3, xy[,2]/1e3, asp=1,
     xlab="Easting Coordinate [km]", ylab="Northing Coordinate [km]",
     pch=20, cex=0.3, col=ifelse(ok3, "red", "blue"))
grid()

So is sf following a different path in instantiating the pipeline? Maybe not?

sf_proj_pipelines(paste0(proj, " type=crs"), paste0(proj0, " type=crs"))

@paleolimbot
Copy link
Contributor

paleolimbot commented Jul 21, 2021

Looks like sf is using proj_create_crs_to_crs() and rgdal is using pipeline.

sf/src/proj.cpp

Line 272 in 370f934

P = proj_create_crs_to_crs(PJ_DEFAULT_CTX, from_to[0], from_to[1], NULL);

...however, I get the same results when using a pipeline operation in sf:

library(sf)
#> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1

phi0_deg <- -20
phi0 <- phi0_deg * pi / 180
proj <- sprintf("+proj=ortho +lat_0=%s +ellips=WGS84", phi0_deg)
proj0 <- "+proj=longlat"

coord_op_inverse <- sf_proj_pipelines(paste0(proj, " +type=crs"), paste0(proj0, " +type=crs"))$definition[1]

grid <- expand.grid(lon=seq(-180,180,2.5), lat=seq(-90,90,2.5))
lonlat <- cbind(grid$lon, grid$lat)
xy <- sf_project(proj0, proj, lonlat, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
LONLAT <- sf_project(proj, proj0, xy, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
LONLAT_pipeline <- sf::sf_project(coord_op_inverse, pts = xy, keep = TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
identical(LONLAT, LONLAT_pipeline)
#> [1] TRUE

ok <- is.finite(LONLAT[,1]) & is.finite(LONLAT[,2])
ok2 <- is.finite(LONLAT_pipeline[,1]) & is.finite(LONLAT_pipeline[,2])

par(mar=c(3,3,1,1), mgp=c(2,0.7,0))
plot(xy[,1]/1e3, xy[,2]/1e3, asp=1,
     xlab="Easting Coordinate [km]", ylab="Northing Coordinate [km]",
     pch=20, cex=0.3, col=ifelse(ok2, "red", "blue"))
grid()

Created on 2021-07-21 by the reprex package (v0.3.0)

@paleolimbot
Copy link
Contributor

I can also confirm that this behaviour occurs in the latest released PROJ (8.1.0)

@paleolimbot
Copy link
Contributor

On closer inspection, the results are identical in rgdal and sf:

library(sf)
#> Linking to GEOS 3.8.1, GDAL 3.2.1, PROJ 7.2.1

phi0_deg <- -20
phi0 <- phi0_deg * pi / 180
proj <- sprintf("+proj=ortho +lat_0=%s +ellips=WGS84", phi0_deg)
proj0 <- "+proj=longlat"

coord_op_inverse <- sf_proj_pipelines(paste0(proj, " +type=crs"), paste0(proj0, " +type=crs"))$definition[1]

grid <- expand.grid(lon=seq(-180,180,2.5), lat=seq(-90,90,2.5))
lonlat <- cbind(grid$lon, grid$lat)
xy <- sf_project(proj0, proj, lonlat, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
LONLAT <- sf_project(proj, proj0, xy, keep=TRUE)
#> Warning in CPL_proj_direct(from_to, as.matrix(pts), keep, warn,
#> authority_compliant): one or more projected point(s) not finite
LONLAT_rgdal  <- rgdal::project(xy, proj, inv=TRUE)
#> Warning in project(bb, proj, inv = inv, use_aoi = FALSE): 2 projected point(s)
#> not finite
#> Warning in rgdal::project(xy, proj, inv = TRUE): 1821 projected point(s) not
#> finite

ok <- is.finite(LONLAT[,1]) & is.finite(LONLAT[,2])
ok2 <- is.finite(LONLAT_rgdal[,1]) & is.finite(LONLAT_rgdal[,2])

identical(ok, ok2)
#> [1] TRUE

Created on 2021-07-21 by the reprex package (v0.3.0)

@rsbivand
Copy link
Member

Correct, I mistyped LONLAT_rgdal <- project(lonlat, proj, inv=TRUE) for LONLAT_rgdal <- project(xy_rgdal, proj, inv=TRUE); I could have added verbose=TRUE to show the pipeline. Yes, rgdal::project() finds the pipeline first then runs it.

@edzer
Copy link
Member

edzer commented Jul 24, 2021

So can we conclude this is a feature of recent PROJ, rather than an issue of sf?

@paleolimbot
Copy link
Contributor

Yes, sf is correctly calling PROJ, where something that was probably not intended is happening. If I get a chance I will open an issue there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants