CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax Deep Learning Ensemble and others #25

cdebom · 2020-07-30T17:40:53Z

We added two models. One bidirectional LSTM and a conv1D.

EiffL · 2020-07-30T18:21:56Z

@cdebom Fantastic! Thank you for your entry :-D Could you document in this thread some of your metric results if you have them?

cdebom · 2020-08-05T13:43:40Z

Dear @EiffL , sorry for the late answer. We perform a couple of tests and retrieve the SNR_3x2 and FOM_3x2 for these two solutions. Here follows a table for the LSTM, later I will send the conv1D.

I updated the previous table since we found some issues in the calculations we had before.

We found, in our tests, SNR_3X2 ~1926 with 10 bins.

cdebom · 2020-08-25T11:40:30Z

I just included the LightGBM model. These are the results we got from it:

Add TCN, AutoML LSTM classifiers, added a custom data loader in AutoML conv1d and Bidirectional LSTM.

Added Deep Ensemble model, including resnet, autolsm, fcn.

cdebom · 2020-08-31T14:03:46Z

Here is summary of our models based on tensorflow/keras our team developed and uploaded:

Ensemble of Deep models
Bidirectional LSTM
Auto ML Bidirectional LSTM
Conv1D NN
AutoML Conv1D NN
Light GBM
Temporal Conv NN

Here are the contributors of our team: Clecio R. Bom (CBPF), Bernardo Fraga (CBPF), Gabriel Teixeira (CBPF), Eduardo Cypriano (USP) and Elizabeth Gonzalez (IATE-CONICET).

Here we optimize for SNR score.

cdebom · 2020-08-31T19:35:22Z

Additionally, we also added a solution based in JAX/FLAX. It is a LSTM that optimizes for SNR score. Thus we upload a total of 8 models.

cdebom · 2020-09-05T00:03:42Z

We added JAX/FLAX Deep Ensemble model. For now we have 7 Tensorflow based entries and 2 NN using JAX/FLAX entries.

This reverts commit 7f73431.

We included a version of challenge.py file that is under bin folder to include the option to run our custom loader.

joezuntz · 2020-09-16T08:58:45Z

Hi @cdebom - thanks for your entries!

I'm currently making sure I can run things properly on my system. There are a couple of imports that I'm now missing

import jax_random as rand
import jax_numpy as jnp

should these be jax.random and jax.numpy?

joezuntz · 2020-09-16T09:09:53Z

Also it looks like at least one file may be missing, lightgbm.py

joezuntz · 2020-09-16T11:06:06Z

And hopefully finally, there are two different classes called ENSEMBLE - in DEEP_ENESMBLE_main.py and Deep_Ensemble_jax.py. Is one of them the right one to use, or should I just rename one so it doesn't clash?

cdebom · 2020-09-16T13:41:06Z

Dear Joe, I am going to check the files, and will come back to you as soon as possible. About the Ensemble, there are two ensemble models. The one in DEEP_ENESMBLE_main.py is based in tensorflow/keras models and optimizes to reduce the classification error. The second ensemble, the Deep_Ensemble_jax.py has a different set of models and it optimizes for SNR and FOM. However I might have uploaded the SNR optimization hardcoded, but it is just a matter of changing jax-metrics for FOM that is already implemented. Em qua., 16 de set. de 2020 às 08:06, joezuntz <[email protected]> escreveu:

…

And hopefully finally, there are two different classes called ENSEMBLE - in DEEP_ENESMBLE_main.py and Deep_Ensemble_jax.py. Is one of them the right one to use, or should I just rename one so it doesn't clash? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHC5UZ6LWWB3BTXTQGMVCSTSGCL25ANCNFSM4POIULUA> .

-- Clécio R. Bom, PhD Assistant Professor / Scientist Brazilian Center for Physics Research clearnightsrthebest.com

joezuntz · 2020-09-17T15:23:09Z

I just realized lightgbm is an external module. I'll install it. And I'll rename the ENSEMBLEs to ENSEMBLE1 and ENSEMBLE2. The challenge has closed now, so it's too late to modify any scientific things.

cdebom · 2020-09-17T17:39:30Z

OK, about the import jax_random as rand import jax_numpy as jnp yes you are correct, it should be jax.random and jax.numpy there must be a typho somewhere. Sorry about that. By the way we also experimented with a custom data loader with different cuts. As we could not pull anything outside tomo_challenge/tomo_challenge/ we leave it in tomo_challenge/tomo_challenge/custom_loader.py and tomo_challenge/tomo_challenge/challenge_custom_loader.py This last file could be used in tomo_challenge/bin/ instead of challenge.py in order to the custom loader work. Em qui., 17 de set. de 2020 às 12:23, joezuntz <[email protected]> escreveu:

…

I just realized lightgbm is an external module. I'll install it. And I'll rename the ENSEMBLEs to ENSEMBLE1 and ENSEMBLE2. The challenge has closed now, so it's too late to modify any scientific things. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHC5UZ5WKTP477G64PLX2VDSGISW5ANCNFSM4POIULUA> .

-- Clécio R. Bom, PhD Assistant Professor / Scientist Brazilian Center for Physics Research clearnightsrthebest.com

joezuntz · 2020-09-22T13:27:33Z

Hi @cdebom - I'm just putting together the full suite of runs we will do. I don't quite see what your custom loader does differently - is it just removing all the galaxies with inf values for the magnitudes? If so I will modify your entries so they just don't provide an estimate for undetected galaxies (which is what inf means)

joezuntz · 2020-09-22T13:32:47Z

And also are you sure you want to do this? In the standard data loader we just set the inf values to 30 instead, so if your problem was with them being inf specifcally that shouldn't be an issue. Possibly we added this feature after you wrote this.

cdebom · 2020-09-22T21:52:10Z

Dear Joe, even setting to 30, we still see some difference in our deep models. For instance, using 10 bins, we found in the tensorflow based Ensemble (resnet, autolstm, fcn) with original loader and jax_metrics: SNR_3x2 : 1920.7 and FOM_3x2 : 10615.2, removing those galaxies in the custom loader we found SNR_3x2 : 1943.4 and FOM_3x2 : 10797.1.

joezuntz · 2020-09-23T09:22:40Z

Okay - I will tweak your code to assign those objects to no bin.

cdebom · 2020-09-23T21:43:50Z

OK, thanks. Just to emphasize that this is something we notice in our models if we remove those galaxies both in training phase and testing phase. So this only makes sense if the training sample dataset has also these galaxies removed.

joezuntz · 2020-09-29T17:45:35Z

When I run the Flax_LSTM code I get this error:

  File "/home/jzuntz/tomo_challenge/bin/../tomo_challenge/classifiers/lstm_flax.py", line 82, in __init__
    self.n_features = options['n_feats']
KeyError: 'n_feats'

What value should n_feats have in the configuration? Is it just the total number of input columns or is it more complicated than that?

cdebom · 2020-09-29T18:03:48Z

Yes, it is the total number of input columns. Cheers Em ter., 29 de set. de 2020 às 14:45, joezuntz <[email protected]> escreveu:

…

When I run the Flax_LSTM code I get this error: File "/home/jzuntz/tomo_challenge/bin/../tomo_challenge/classifiers/lstm_flax.py", line 82, in __init__ self.n_features = options['n_feats'] KeyError: 'n_feats' What value should n_feats have in the configuration? Is it just the total number of input columns or is it more complicated than that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHC5UZ7EDS4VK4YDXXO4YE3SIIMM5ANCNFSM4POIULUA> .

-- Clécio R. Bom, PhD Assistant Professor / Scientist Brazilian Center for Physics Research clearnightsrthebest.com

joezuntz · 2020-09-29T18:10:14Z

Why is that a configuration input? Why is it not calculated from the data?

cdebom · 2020-10-14T22:23:25Z

Just for the record, here are some of the results we had with 10 bins. Since training using jax was very computer intensive we tested using 10% of the data in a couple random choices of training set. To have a fair comparison with the other methods we also trained with 10%.

It is interesting to note that in the couple of runs of jax LSTM (optimized for SNR) there were variations in SNR_3X2 and FOM_3X2. However it seems that there is a trade off between those two, when one gets a bit higher the other gets a bit lower.

This is the binning plot for JAX autolstm run2

cdebom added 2 commits July 30, 2020 12:59

Add files via upload

f0a34a5

Add files via upload

49e0388

EiffL added the entry Challenge entry label Jul 30, 2020

cdebom added 2 commits August 24, 2020 10:16

Add a simple LightGBM classifier.

c9b467c

We found, in our tests, SNR_3X2 ~1926 with 10 bins.

I corrected the file's description

d42091f

cdebom changed the title ~~LSTM and conv1D models~~ LSTM, Conv1D, LightGBM models Aug 24, 2020

cdebom added 3 commits August 31, 2020 10:32

Added new models, added loader to others

dd086e2

Add TCN, AutoML LSTM classifiers, added a custom data loader in AutoML conv1d and Bidirectional LSTM.

Added Deep Ensemble Model

88de451

Added Deep Ensemble model, including resnet, autolsm, fcn.

Added Keras contrib

ce001b2

cdebom changed the title ~~LSTM, Conv1D, LightGBM models~~ CBPF IA Team models Aug 31, 2020

cdebom added 2 commits August 31, 2020 14:33

small correction

7e6a7c4

Jax/Flax based LSTM

16733c9

Here we optimize for SNR score.

cdebom changed the title ~~CBPF IA Team models~~ CBPF IA Team models LSTM, Deep Ensemble, Flax NN an others Sep 1, 2020

cdebom changed the title ~~CBPF IA Team models LSTM, Deep Ensemble, Flax NN an others~~ CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax NN an others Sep 1, 2020

cdebom changed the title ~~CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax NN an others~~ CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax NN and others Sep 1, 2020

Jax/Flax Deep Ensemble

2b80318

cdebom changed the title ~~CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax NN and others~~ CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax Deep Learning Ensemble and others Sep 5, 2020

cdebom added 7 commits September 14, 2020 22:03

Adding example of some classifiers

4f2d249

Fixes in the loader

1d45a33

Add files via upload

ae7817c

Add custom data loader.

a4b9512

Add a custom data loader option.

7f73431

Revert "Add a custom data loader option."

1536488

This reverts commit 7f73431.

version of challenge.py to include the custom loader

3aa3a47

We included a version of challenge.py file that is under bin folder to include the option to run our custom loader.

cdebom added 10 commits September 15, 2020 02:20

Small Fix in JAX Deep Ensemble

22f342e

import fix in TF resnet

e177e8f

Update de_resnet.py

39b407b

Update de_resnet.py

0cc731a

import fix in Ensemble LSTM

c74df53

Update de_autolstm.py

18566b9

Update de_resnet.py

12ca8c7

Update de_fcn.py

596eff7

import fix

39dad65

import fix

c9a5cd1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax Deep Learning Ensemble and others #25

CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax Deep Learning Ensemble and others #25

cdebom commented Jul 30, 2020

EiffL commented Jul 30, 2020

cdebom commented Aug 5, 2020 •

edited

Loading

cdebom commented Aug 25, 2020 •

edited

Loading

cdebom commented Aug 31, 2020 •

edited

Loading

cdebom commented Aug 31, 2020

cdebom commented Sep 5, 2020

joezuntz commented Sep 16, 2020

joezuntz commented Sep 16, 2020

joezuntz commented Sep 16, 2020

cdebom commented Sep 16, 2020 via email

joezuntz commented Sep 17, 2020

cdebom commented Sep 17, 2020 via email •

edited

Loading

joezuntz commented Sep 22, 2020 •

edited

Loading

joezuntz commented Sep 22, 2020

cdebom commented Sep 22, 2020

joezuntz commented Sep 23, 2020

cdebom commented Sep 23, 2020

joezuntz commented Sep 29, 2020

cdebom commented Sep 29, 2020 via email

joezuntz commented Sep 29, 2020

cdebom commented Oct 14, 2020

CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax Deep Learning Ensemble and others #25

Are you sure you want to change the base?

CBPF IA Team models LSTM, Deep Ensemble, Jax/Flax Deep Learning Ensemble and others #25

Conversation

cdebom commented Jul 30, 2020

EiffL commented Jul 30, 2020

cdebom commented Aug 5, 2020 • edited Loading

cdebom commented Aug 25, 2020 • edited Loading

cdebom commented Aug 31, 2020 • edited Loading

cdebom commented Aug 31, 2020

cdebom commented Sep 5, 2020

joezuntz commented Sep 16, 2020

joezuntz commented Sep 16, 2020

joezuntz commented Sep 16, 2020

cdebom commented Sep 16, 2020 via email

joezuntz commented Sep 17, 2020

cdebom commented Sep 17, 2020 via email • edited Loading

joezuntz commented Sep 22, 2020 • edited Loading

joezuntz commented Sep 22, 2020

cdebom commented Sep 22, 2020

joezuntz commented Sep 23, 2020

cdebom commented Sep 23, 2020

joezuntz commented Sep 29, 2020

cdebom commented Sep 29, 2020 via email

joezuntz commented Sep 29, 2020

cdebom commented Oct 14, 2020

cdebom commented Aug 5, 2020 •

edited

Loading

cdebom commented Aug 25, 2020 •

edited

Loading

cdebom commented Aug 31, 2020 •

edited

Loading

cdebom commented Sep 17, 2020 via email •

edited

Loading

joezuntz commented Sep 22, 2020 •

edited

Loading