ANY23-321 Add openie toggle functionality to service #56

lewismc · 2018-01-03T00:13:27Z

This issue is a large step towards addressing https://issues.apache.org/jira/browse/ANY23-321
I am however not able to currently register the OpenIEExtractor within the ExtractorGroup/Factory when undertaking an extractor. I've debugged this down to the code in the URLClassLoader.addURL of Any23PluginManager. The OpenIE JAR's are dynamically loaded however the Extractor implementation does not seem to be registered when the extraction is executed.

@ansell if you are able to pull this code and debug it would be greatly appreciated. I have specific lines for debugging if this would be helpful. Thank you in advance for any assistance here.

P.S. you will also see that I've been making attempts to update the plugin documentation.

P.P.S I actually remember encountering a similar issue previously when attempting to register a plugin via the command line... I think dynamic ClassLoading is broken in Any23 right now. I am keen to fix it so any help here is appreciated folks.

lewismc · 2018-02-03T05:56:34Z

@HansBrende @jgrzebyta can you both please have a look at dynamic plugin loading in this patch? I am at a loss, having tried to debug this with no progress. Thanks in advance,

HansBrende · 2018-02-03T07:09:37Z

@lewismc you said you had some specific lines for debugging?

lewismc · 2018-02-03T08:02:57Z

Thanks @HansBrende :)
Once you have built the Webservice and deployed to Tomcat, you should make sure that you check the openie checkbox in the GUI.
You can set a breakpoint at the openie conditional toggle on in Servlet.java which will enable you to see that the parameter is being received on server side. From here, you need to look into how the Any23PluginManager does dynamic classloading.
Let me know how you get on and I can provide a bit more context as to how the directory openie structure works for dynamic loading.
Thanks in advance,

Signed-off-by: Jacek Grzebyta <[email protected]>

HansBrende · 2018-02-04T04:02:50Z

@lewismc I really have no idea how to deploy to Tomcat, being a bit of a noob, but have you tried doing something like this in Servlet.java?

if (loadedJars) { 
    ExtractorRegistry r = ExtractorRegistryImpl.getInstance();
    pManager.getExtractors().forEachRemaining(r::register);
    ...
}

jgrzebyta · 2018-02-05T10:59:36Z

@lewismc

I have: Failed to execute goal on project apache-any23-service: Could not resolve dependencies for project org.apache.any23:apache-any23-service:war:2.3-SNAPSHOT: Failure to find org.allenai.openie:openie_2.11:jar:4.2.6 in https://svn.apache.org/repos/asf/any23/repo-ext/ was cached in the local repository, resolution will not be reattempted until the update interval of any23-repository-external has elapsed or updates are forced.

~~IMHO it shouldn't be said in the apache-any23-service pom that the dependency is provided. Provided means it comes together with tomcat.~~ It did not help. However I found this jar in my local repo.

@HansBrende did you had that problem as well? I have it also when I exec mvn in command line.

HansBrende · 2018-02-05T14:07:15Z

@jgrzebyta Yes, I had that problem too. Mvn build fails, but that's not related to the classloader problem @lewismc is having (which my comment above should fix--the extractor registry automatically registers classes only via the system classloader, so extractors that are loaded by other means have to be explicitly registered).

HansBrende · 2018-02-08T19:39:39Z

@lewismc have you been able to test whether or not my suggested insertion fixes your problem?

lewismc · 2018-02-08T19:53:29Z

@HansBrende not yet sorry, I have been overloaded :( I will try tonight.

ansell · 2018-02-08T21:38:55Z

I don't have time right now to debug this or work on its design. It isn't a pattern I have worked on in the past. All ServiceLoader projects that I have worked on in the past either have all of the service Factory objects initialised as startup, or they do periodic scans, but always create and initialise the service Factory objects whenever they are discovered.

Is this issue attempting to delay/extract the OpenIE initialisation by delaying/postponing the Factory initialisation? If so, the focus could possibly be on making the OpenIE initialisation lazy to only occur when the Factory is actually required/used to create an object instance.

A more configurable lifecycle possibly requires OSGi, which I again have absolutely no experience with, but from the little understanding each time I look, it appears to allow user driven initialisation (push rather than pull, if that makes sense).

…nient

lewismc · 2018-02-23T18:02:33Z

Hi @HansBrende your suggestion fixed the code so I have pushed for further review.
@ansell thank you for the suggestions I think you are absolutely right... I've posted a question on users@maven along with my followup so hopefully I can keep the service pom clean by obtaining the correct solution.
I'll post here once I know. Thanks all.

lewismc · 2018-02-23T18:07:24Z

I've messed this PR up now so I will close and reopen another once I have the dependency issue resolved. I'll also provide an update to the documentation as to how dynamic classloading is done per example.

lewismc · 2018-02-24T01:58:55Z

OK folks, so this issue is now available for testing. I can see two immediate issues

OpenIE is memory-intensive and can prety easily exhaust >6GB memory for larger jobs. This is an issue if we are running this as a service via any23.org,
There is a considerable amount of bloat on the resulting service artifacts e.g. WAR, tarballs, zip files, etc. See below

lmcgibbn@LMC-056430 /usr/local/any23/service/target(ANY23-321) $ ls -lh
total 13499344
drwxr-xr-x  5 lmcgibbn  wheel   170B Feb 23 17:47 apache-any23-service-2.2-SNAPSHOT
-rw-r--r--  1 lmcgibbn  wheel   851M Feb 23 17:50 apache-any23-service-2.2-SNAPSHOT-server-embedded.tar.gz
-rw-r--r--  1 lmcgibbn  wheel   851M Feb 23 17:50 apache-any23-service-2.2-SNAPSHOT-server-embedded.zip
-rw-r--r--  1 lmcgibbn  wheel   846M Feb 23 17:48 apache-any23-service-2.2-SNAPSHOT-with-deps.tar.gz
-rw-r--r--  1 lmcgibbn  wheel   846M Feb 23 17:49 apache-any23-service-2.2-SNAPSHOT-with-deps.zip
-rw-r--r--  1 lmcgibbn  wheel   784M Feb 23 17:49 apache-any23-service-2.2-SNAPSHOT-without-deps.tar.gz
-rw-r--r--  1 lmcgibbn  wheel   783M Feb 23 17:48 apache-any23-service-2.2-SNAPSHOT-without-deps.war
-rw-r--r--  1 lmcgibbn  wheel   784M Feb 23 17:49 apache-any23-service-2.2-SNAPSHOT-without-deps.zip
-rw-r--r--  1 lmcgibbn  wheel   846M Feb 23 17:47 apache-any23-service-2.2-SNAPSHOT.war
drwxr-xr-x  2 lmcgibbn  wheel    68B Feb 23 17:47 archive-tmp
drwxr-xr-x  4 lmcgibbn  wheel   136B Feb 23 17:47 classes
drwxr-xr-x  3 lmcgibbn  wheel   102B Feb 23 17:47 generated-sources
drwxr-xr-x  3 lmcgibbn  wheel   102B Feb 23 17:47 generated-test-sources
drwxr-xr-x  3 lmcgibbn  wheel   102B Feb 23 17:47 maven-archiver
drwxr-xr-x  3 lmcgibbn  wheel   102B Feb 23 17:47 maven-status
drwxr-xr-x  4 lmcgibbn  wheel   136B Feb 23 17:47 test-classes
drwxr-xr-x  4 lmcgibbn  wheel   136B Feb 23 17:47 war-legals

lewismc · 2018-02-27T03:06:25Z

Any comments here folks?

HansBrende · 2018-02-27T03:13:27Z

I mean, I'm not sure what kind of resources you want your webservice to be using, but 6GB memory sounds like a blocker to me. Using OpenIE with the core api would be opt-in only, right?

lewismc · 2018-02-27T03:54:35Z

Yes this is optizonal for the core API and would really only be used to demonstrate applications that can be built using Any23’s dynamic plugin loading capabilities. This is a good excuse to update he Documentation regardless of whether we decide to merge it into master. Did you try it locally and see what kind of results you get? It’s pretty cool and would nicely complement something like Jena’s full text search https://jena.apache.org/documentation/query/text-query.html Relistically the triples which are extracted using OpenIE are free text, not defined vocabulary hence personally I think it highlights quite a nice use case.

On Mon, Feb 26, 2018 at 19:13 Hans Brende ***@***.***> wrote: I mean, I'm not sure what kind of resources you want your webservice to be using, but 6GB memory sounds like a blocker to me. Using OpenIE with the core api would be opt-in only, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#56 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABHJlxLAaAzyXR3gLi0Gm5QRjEzBUl5Tks5tY3LZgaJpZM4RROYY> .

-- http://home.apache.org/~lewismc/ http://people.apache.org/keys/committer/lewismc

HansBrende · 2018-02-27T04:59:43Z

I haven't tried it locally, but I agree it's pretty cool functionality and would aptly demonstrate that plugin-loading works.

My main concern, if this will be deployed to any23.org, would be the RAM usage.

One solution might be to add a configuration flag that either enables or disables OpenIE processing (the OpenIE toggle would only be displayed if the flag were enabled)--and disable the flag by default. (Unless whoever's paying for the any23.org web service doesn't mind each request using 6 GB--in that case I guess it wouldn't matter.)

HansBrende · 2018-02-27T05:13:59Z

(The flag could even be read at the build-level: for example, you might add a build step that reads your configuration file and modifies the HTML to show/enable or hide/disable the toggle according to whether or not the OpenIE flag is enabled--so as not to make an unnecessary web request. Of course on the server side, you could just read the config file in on the fly--and respond with an error if the user requested OpenIE, but it was disabled in the config. I personally have no experience with building complex build tasks, so I have no idea whether or not this is feasible, or even a good idea. But just a thought.)

lewismc · 2018-02-27T06:02:01Z

My main concern, if this will be deployed to any23.org, would be the RAM usage.

We can always request more RAM for the dedicated VM powering any23.org

One solution might be to add a configuration flag that either enables or disables OpenIE processing (the OpenIE toggle would only be displayed if the flag were enabled)--and disable the flag by default.

This is exactly whats provided in the patch. I guessed the it should be disabled by default. You can see examples of the toggle functionality in the snapshot's below

Note the new OpenIE toggle box above

The above information is displayed when you click on the '?' icon.

HansBrende · 2018-02-27T06:12:13Z

@lewismc my suggestion was to provide a config flag in addition to the existing checkbox (the difference being: the config flag would not be controlled by the user accessing the webpage--but by the server itself, in order to specify whether or not the server allows the user to switch OpenIE "on".)

(It would have been just a way to leave the plugin-loading code there so that people could look it over and test it out locally by changing the config flag, while not actually allowing them to do so on the any23 webservice.)

But if the RAM usage is not a problem for any23.org, then there's no need for it.

HansBrende · 2018-02-27T06:47:28Z

In lieu of the config flag, it might be nice to just handle OutOfMemoryErrors gracefully. Maybe something like this in OpenIEExtractor.run()?

} catch (OutOfMemoryError e) {
    //let the gc do its thang
    openIE = null;
    out.notifyIssue(IssueReport.IssueLevel.FATAL, "Not enough memory available to perform OpenIE extraction.", -1, -1);
    return;
}

In any case, if any23.org can handle 6 GB of memory per instance (or rather, 10 GB, the -Xmx value recommended by the OpenIE readme), then I say +1. Cool functionality, and it would be great to be able to use it in the any23 webservice!

HansBrende · 2018-02-27T14:34:02Z

On a slightly tangential topic, according to this github page, the current version of OpenIE is OpenIE 5, located here, whereas it looks like we are using OpenIE 4.2.6.

lewismc · 2018-02-27T15:27:54Z

I logged an issue in JIRA about this. I’ve also been over to see that disappointingly there is not Maven artifact available for OpenIE 5. I didn’t hear anything back from the OpenIE team either dair-iitd/OpenIE-standalone#6 I suppose one of us could go and contribute the module, however at that time, I believe I just started working on the Any23 Service + OpenIE integration. As I said it was a while ago.

On Tue, Feb 27, 2018 at 06:34 Hans Brende ***@***.***> wrote: On a slightly unrelated topic, according to this github page <https://github.com/knowitall/openie>, the current version of OpenIE is OpenIE 5, located here <https://github.com/dair-iitd/OpenIE-standalone>, whereas it looks like we are using OpenIE 4.2.6. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#56 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABHJl-Ek4C9mQYf7-23oCUzOq8erMT21ks5tZBJegaJpZM4RROYY> .

-- http://home.apache.org/~lewismc/ http://people.apache.org/keys/committer/lewismc

HansBrende · 2018-02-27T15:49:53Z

Ah, I see. Well, it may not be the end of the world if we stick with 4.2.6 for now--looks like that version requires less memory anyway.

What do you think about my OOM handling suggestion? I edited my comment to provide some sample code.

lewismc · 2018-02-27T18:13:58Z

@HansBrende

What do you think about my OOM handling suggestion? I edited my comment to provide some sample code.

Excellent suggestion, I've accommodated it. I also agree that another setting, possibly within core Any23 Configuration would be good, however i think that this can be added if the exception handling is not sufficient. That would be a different issue.

lewismc and others added 6 commits January 3, 2018 00:05

ANY23-321 Add openie toggle functionality to service

706e891

Merge branch 'master' into ANY23-321

b71142f

Merge branch 'master' into ANY23-321

6660ed8

Merge branch 'master' into ANy23-321

4f40fe0

ANY23-327 Change log level to debug for RDFUtils.isAbsoluteIRI()

c2ddb58

ANY23-321 Add openie toggle functionality to service

63ffc9e

Merge remote-tracking branch 'HansBrende/ANY23-327'

5a3ba9d

Signed-off-by: Jacek Grzebyta <[email protected]>

HansBrende and others added 9 commits February 8, 2018 23:27

ANY23-264 Upgrade to use public commons-csv instead of custom SNAPSHOT

7a7db20

Update poweredby page to account for Nutch usage

616bdf8

Remove external repositories from pom.xml

c1679ce

Revert Any23 RC#1

4131761

ANY23-328 Strip comments from json-ld content to make parsing more le…

189bf26

…nient

Revert to 2.2

5d3d715

Merge branch 'ANY23-328' of https://github.com/HansBrende/any23

778d05e

Merge into master

073190b

ANY23-321 Add openie toggle functionality to service

69109f3

ANY23-321 Add openie toggle functionality to service

b3806d3

ANY23-321 Add openie toggle functionality to service

71bf171

asfgit merged commit 71bf171 into apache:master Feb 28, 2018

lewismc deleted the ANY23-321 branch February 28, 2018 04:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ANY23-321 Add openie toggle functionality to service #56

ANY23-321 Add openie toggle functionality to service #56

lewismc commented Jan 3, 2018

lewismc commented Feb 3, 2018

HansBrende commented Feb 3, 2018

lewismc commented Feb 3, 2018

HansBrende commented Feb 4, 2018

jgrzebyta commented Feb 5, 2018 •

edited

Loading

HansBrende commented Feb 5, 2018 •

edited

Loading

HansBrende commented Feb 8, 2018

lewismc commented Feb 8, 2018

ansell commented Feb 8, 2018

lewismc commented Feb 23, 2018

lewismc commented Feb 23, 2018

lewismc commented Feb 24, 2018

lewismc commented Feb 27, 2018

HansBrende commented Feb 27, 2018

lewismc commented Feb 27, 2018 via email

HansBrende commented Feb 27, 2018

HansBrende commented Feb 27, 2018 •

edited

Loading

lewismc commented Feb 27, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading

lewismc commented Feb 27, 2018 via email

HansBrende commented Feb 27, 2018

lewismc commented Feb 27, 2018

ANY23-321 Add openie toggle functionality to service #56

ANY23-321 Add openie toggle functionality to service #56

Conversation

lewismc commented Jan 3, 2018

lewismc commented Feb 3, 2018

HansBrende commented Feb 3, 2018

lewismc commented Feb 3, 2018

HansBrende commented Feb 4, 2018

jgrzebyta commented Feb 5, 2018 • edited Loading

HansBrende commented Feb 5, 2018 • edited Loading

HansBrende commented Feb 8, 2018

lewismc commented Feb 8, 2018

ansell commented Feb 8, 2018

lewismc commented Feb 23, 2018

lewismc commented Feb 23, 2018

lewismc commented Feb 24, 2018

lewismc commented Feb 27, 2018

HansBrende commented Feb 27, 2018

lewismc commented Feb 27, 2018 via email

HansBrende commented Feb 27, 2018

HansBrende commented Feb 27, 2018 • edited Loading

lewismc commented Feb 27, 2018 • edited Loading

HansBrende commented Feb 27, 2018 • edited Loading

HansBrende commented Feb 27, 2018 • edited Loading

HansBrende commented Feb 27, 2018 • edited Loading

lewismc commented Feb 27, 2018 via email

HansBrende commented Feb 27, 2018

lewismc commented Feb 27, 2018

jgrzebyta commented Feb 5, 2018 •

edited

Loading

HansBrende commented Feb 5, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading

lewismc commented Feb 27, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading

HansBrende commented Feb 27, 2018 •

edited

Loading