Fermi #199

shb46 · 2024-09-20T17:24:56Z

Fermi GBM Schema for a single schema, "ALERT" - Initial Version

gcn/notices/fermi/gbm/FermiGBMAlert.schema.json

Vidushi-GitHub · 2024-10-02T13:46:15Z

gcn/notices/fermi/gbm/strict/FermiGBMAlert.schema.json

+      "type": "string",
+      "description": "Date and time of notice creation [UTC, ISO 8601], ex YYYY-MM-DDTHH:MM:SS.ssssssZ"
+    },
+    "alert_tense": {


It's already present in core schema, you just have to ref the schema.
Same for most of the properties below, rate_snr, trigger_time etc.

Vidushi-GitHub · 2024-10-02T13:48:47Z

We have planned a mission page, where you can keep the content present at readme file.

Vidushi-GitHub · 2024-10-02T13:58:36Z

Hi @shb46, sorry for the delayed response due to travel and vacation. Most of the stuff looks fine.
My main confusion about schema is why strict schema there and why you are re-writing core properties, instead of ref?

shb46 · 2024-10-02T21:27:59Z

Hi @Vidushi-GitHub,

Hope everything is well! Here are my replies:

We have planned a mission page, where you can keep the content present at readme file.

Can we have an instrument page as well? the lat and the GBM instruments are run by separate teams.
Can notice type have its own page?

Hi @shb46, sorry for the delayed response due to travel and vacation. Most of the stuff looks fine. My main confusion about schema is why strict schema there and why you are re-writing core properties, instead of ref?

Those are two independent schemas intended to describe the same notice type:

The "Kafka" one: fermi/gbm/FermiGBMAlert.schema.json
- Makes $ref-s to the core schema.
- Doesn't $ref the strict schema.
- Validates new notices received by the Kafka server.
- More permissive compared to the strict schema.
The "strict" schema: fermi/gbm/strict/FermiGBMAlert.schema.json:
- Doesn't $ref the core schema, nor the "Kafka" schema.
- Is less permissive than the Kafka schema.
- Clarifies what to expect in a notice, incl. which core fields are not used.
- Used by the producer, in addition to the Kafka schema.

[Edit:] I should rename the strict schema file so it has a different name.

Vidushi-GitHub · 2024-10-10T19:46:31Z

Hi Boyan,
Could you please change the filename?
fermi/gcn/notices/fermi/gbm/FermiGBMAlert.schema.json to fermi/gcn/notices/fermi/gbm/alert.schema.json?

As it's already in fermi/gbm folder, again fermiGBM is redundant.

Vidushi-GitHub · 2024-10-10T19:56:26Z

We have planned a mission page, where you can keep the content present at readme file.

Can we have an instrument page as well? the lat and the GBM instruments are run by separate teams.

Can notice type have its own page?

For Fermi mission page, we have information for GCN Classic Notices at https://gcn.nasa.gov/missions/fermi, and would move forward with updating it.
@jracusin shall we separate out GBM, LAT content into two mission pages as available at https://gcn.nasa.gov/missions/fermi Or, just update the existing one as single page?

Vidushi-GitHub · 2024-10-10T20:12:04Z

Those are two independent schemas intended to describe the same notice type:

The "Kafka" one: fermi/gbm/FermiGBMAlert.schema.json

Makes $ref-s to the core schema.

Doesn't $ref the strict schema.

Validates new notices received by the Kafka server.

More permissive compared to the strict schema.

The "strict" schema: fermi/gbm/strict/FermiGBMAlert.schema.json:

Doesn't $ref the core schema, nor the "Kafka" schema.

Is less permissive than the Kafka schema.

Clarifies what to expect in a notice, incl. which core fields are not used.

Used by the producer, in addition to the Kafka schema.

[Edit:] I should rename the strict schema file so it has a different name.

I am trying to understand why do you wanna create two schemas for same notice types? What is advantage of not referring $ref or kafka schema? Would you produce same notices from two schemas?
Definitions/clarifications shouldn't be problem, that can be added in alert schema as well.

@lpsinger, @jracusin do you have comments on the same notices from different schemas, alert ($ref to core) and strict (own definitions)?

shb46 · 2024-10-10T21:17:20Z

@jracusin:
Re. separate GBM/LAT pages.
Please let me know if it is possible. If yes, we should get the consensus of the GBM and LAT teams before we make a change. I think it will work with or without a "mission" page, i.e. either one of:

Missions

Fermi

GBM

LAT

or

Missions

FermiGBM

FermiLAT

shb46 · 2024-10-10T21:17:31Z

@Vidushi-GitHub: File renamed. The JPG and MD files removed.

shb46 · 2024-10-11T01:17:54Z

@jracusin: P.S.: I'm not planning to change the mission page at this time.

dakota002 · 2024-10-11T14:45:42Z

Hi @shb46 , I have a couple additional requests for this update:

Can you rename the strict schema and examples as well:

gcn/notices/fermi/gbm/strict/FermiGBMStrictAlert.schema.json -> gcn/notices/fermi/gbm/strict/alert.schema.json
gcn/notices/fermi/gbm/strict/FermiGBMAlert1.example.json -> gcn/notices/fermi/gbm/strict/alert.FermiGBMAlert1.example.json
gcn/notices/fermi/gbm/strict/FermiGBMAlert2.example.json -> gcn/notices/fermi/gbm/strict/alert.FermiGBMAlert2.example.json
gcn/notices/fermi/gbm/strict/FermiGBMStrictAlert1.example.json -> gcn/notices/fermi/gbm/strict/alert.FermiGBMStrictAlert1.example.json
gcn/notices/fermi/gbm/strict/FermiGBMStrictAlert2.example.json -> gcn/notices/fermi/gbm/strict/alert.FermiGBMStrictAlert2.example.json

The schema name change reason is the same as the other schema as @Vidushi-GitHub mentioned, as the path already gives the name. As for the examples, the pattern of: alert.[any specific name for the example].example.json is recognized in our schema browser and parsers, and will help automate our docs.

Once this is all set, could you then squash your commits down to 1 single commit? I can assist in this if you have questions before doing so.

shb46 · 2024-10-11T15:50:20Z

Hi, @dakota002,

Please check the new file names before squashing.

dakota002 · 2024-10-11T17:36:11Z

Hi, @dakota002,

Please check the new file names before squashing.

The names look good to me!

shb46 · 2024-10-11T18:48:21Z

@dakota002: Did I squash it enough?

dakota002 · 2024-10-11T19:50:59Z

@dakota002: Did I squash it enough?

I don't believe so, I see 52 commits now. You may have pulled the history back into your branch before pushing it out. You should just run the command git push -f to assert that the new history is correct

shb46 · 2024-10-14T14:59:40Z

@dakota002: Can't figure it out. I'm going to need your help with this.

gcn/notices/fermi/gbm/strict/alert.2.example.json

…mber with which the units go together."

Vidushi-GitHub · 2024-11-06T03:12:20Z

gcn/notices/fermi/gbm/strict/alert.1.example.json

+    "on"
+  ],
+  "event_name": ["GRB240824A"],
+  "hitl": false,


Please make the example field name consistent with schema.
Additionally, validate the schema against examples, in case any change is left out.

@Vidushi-GitHub: Changed. Thank you for the reminder!

Vidushi-GitHub

@jracusin please have a look, and provide your feedback/approval.

jracusin · 2024-11-16T10:40:43Z

gcn/notices/fermi/gbm/alert.schema.json

+  "properties": {
+    "notice_type": {
+      "type": "string",
+      "description": "Notice title"


Can this be an enum? How many options are there for notice title? Are you always going to refer back to GCN Classic notice types? You don't need to be tied to those.

Added "const": "FERMI_GBM_ALERT". This is redundant with $schema and the topic. Thanks for pointing out.

jracusin · 2024-11-16T10:42:37Z

gcn/notices/fermi/gbm/alert.schema.json

+    },
+    "human_in_the_loop": {
+      "type": "boolean",
+      "description": "Someimes abbreviated as \"HITL\".  When false, the notice was generated entirely automatically.  A value of true means that a human had an input, while the notice was being prepared.  The latter includes notices, which weren't altered by the human input, but a human was given the option to make an input, or asked to confirm, or approve before further processing."


Is HITL also a notice type? This description could also be much more concise. For example:
"True when a human had input or confirmed before further processing; False when the process was fully automated."

Shorter and better. Thanks for the suggestion.

jracusin · 2024-11-16T10:44:42Z

gcn/notices/fermi/gbm/alert.schema.json

+    },
+    "trigger_algorithm_number": {
+      "type": "number",
+      "description": "The number identifying the algorithm that detected the gamma-ray burst and caused the trigger.  \"Trigger algorithm\" here is a moniker for a method or a criterion that watches for significant rate increase in the observed rate.  Multiple \"trigger algorithms\" are active simultaneously and any one can trigger independently.  The trigger_algorithm_number, tells which one did."


Are these trigger algorithms described somewhere in a document or paper. Can you just reference that document with something like the following. There's too much extra information in this description.
"The instrument flight software algorithm that triggered on this event, dependent on timescale, energy, and significance as described in ...".

jracusin · 2024-11-16T10:46:25Z

gcn/notices/fermi/gbm/alert.example.json

+  "id": ["745121685", "bn240812094"],
+  "instrument": "GBM",
+  "latitude": 23.83,
+  "light_curve_url": "http://heasarc.gsfc.nasa.gov/FTP/fermi/data/gbm/triggers/2024/bn240812094/quicklook/glg_lc_medres34_bn240812094.gif",


These don't need to be alphabetical, and it seems strange to separate latitude and longitude. In general, these examples should be in some logical order.

Alphabetical order helps bookkeeping. I'd like to keep it this way.

jracusin · 2024-11-16T10:50:10Z

gcn/notices/fermi/gbm/detectors.schema.json

+    "detectors_status": {
+      "type": "array",
+      "items": {
+        "$ref": "../../core/DetectorStatus.schema.json"


These can be referenced by name rather than indexed, which would then not require this detectors schema at all. See https://github.com/nasa-gcn/gcn-schema/blob/main/gcn/notices/burstcube/alert.example.json or https://github.com/nasa-gcn/gcn-schema/blob/main/gcn/notices/glowbug/Alert.schema.json, which you previously came up with.

I think we are discussing two design points here:

Array vs. names;

To $ref or not to $ref the core/DetectorStatus.

Thanks for suggesting an alternative to the array. The two choices were discussed at our pipeline meeting (1 vote for names) and the IOTL meeting. The 14-array won. Consider that it is a bit mask in the data products, in which the bits are indexed 0-13. The same is in the classic notes, incl. the text ones. Also the old and the new GBM tools use straight arrays. In retrospect, I should had used an array for Glowbug as well.

On point (2), if you would like to remove the $ref, I'll be happy to replace it with the enum. just let me know.

Hi @shb46, Could you please implement the detectors schema within alert.schema.json?
I understand the logic of making it a library and calling it. But it's just a few lines field and would make alert.schema.json look like a self-consistent schema.

Are you planning to add more topics in the future and wanna call detectors.schema.json?

@Vidushi-GitHub: The detectors schema, if we keep it, will be shared with another topic. Do you still want me to make it inline?

@shb46 Ok, thanks. If you plan to have more topics related to Fermi/GBM, then that's fine.
Please remember to limit topics per mission to simplify user subscriptions to new Notices. The new Schema allows multiple Notices using a single Schema since fields are not mandatory.

@Vidushi-GitHub: Our intent was for GBM to have 5 topics. Hope that is not too much. I suspect we might need a 6th one. Separately, there might be topics for internal distribution only but this hasn't been discussed yet.

My plan is to have a 1-to-1-to-1 correspondence of classic notice types, schemas and topics. I think it is a little too late to steer towards making a single schema. We are ahead with approving the ALERT schema and it will set us back. My deadline is Dec 12.

gcn/notices/fermi/gbm/strict/alert.schema.json

jracusin · 2024-11-16T10:52:34Z

gcn/notices/fermi/gbm/strict/alert.schema.json

+  "$id": "https://gcn.nasa.gov/schema/main/gcn/notices/fermi/gbm/strict/alert.schema.json",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "GCN Schema for Fermi GBM Alert Notices",
+  "description": "This schema is a stricter version of the more permissive main schema, https://gcn.nasa.gov/schema/main/gcn/notices/fermi/gbm/alert.schema.json.  The Kafka server uses the main schema for new notices.  Each Alert notice (classic numeric type 110) must validate against the main schema and should also validate against this one.  Examples in this directory come in two versions referencing the main and the strict schemas respectively.  The intent of the strict schema is to help understanding the Alert notice, since it better describes what content to be expected.",


I still don't understand the purpose or use of this strict schema. Each schema is a Kafka topic, though you are welcome to multiple examples. The GBM topic will be gcn.notices.fermi.gbm.alert, so what is the use case for gcn.notices.fermi.gbm.strict.alert?

I wasn't planning on having a gcn.notices.fermi.gbm.strict.alert topic, only the gcn.notices.fermi.gbm.alert. The same way the server functionality doesn't depend on the examples, it should ignore the strict schemas. If the server does notice validation, it should use only the .../gbm/alert.schema.json. All I want the strict schema to do, is will just be there in addition to the examples and will serve a similar illustrative/documentation purpose.

The JSON objects described by the strict schema are a subset of the objects described by the "non-sctrict" one. If, by looking at the examples, one has doubts about what to expect, then the strict schema will help clear it up. This will become more so with the other notice types.

@shb46, I looked at a diff between gcn/notices/fermi/gbm/alert.schema.json and gcn/notices/fermi/gbm/strict/alert.schema.json. It looks like the only difference is that in the former, you include types from core schema, whereas in the latter, you hardcode the definitions of the fields from the core schema that you are actually using. Please don't do that! Please remove the "strict" schema directory from your PR.

I think that this exposes an interesting issue, that when you include a core schema, it is not clear to users which of its many optional fields you actually intend to populate. That affects all GCN notice types, not just Fermi. @jracusin, let's open another issue to study that problem and come up with options.

@jracusin, @lpsinger: Done.

IMHO, having up to a web page per topic (notice type) would help with the issue. For Fermi GBM it is also possible to have a single page with a field vs. notice type matrix.

jracusin · 2024-11-16T10:53:45Z

gcn/notices/fermi/gbm/strict/alert.2.example.json

@@ -0,0 +1,34 @@
+{
+  "$schema": "https://gcn.nasa.gov/schema/main/gcn/notices/fermi/gbm/alert.schema.json",
+  "additional_info": "*** ABOUT THIS EXAMPLE: This is an example of a Fermi GBM Alert GCN notice. Only stricly necessary fields are present. This 'about' note is not going to be part of streamed/real notices. This field may be missing or empty in case of no comments. An example of a typical notice comment follows. ***\n\nLightcurve and other quicklook files will not be created until about 15 minutes after the trigger.",


What is different between these examples, please add some info about their difference rather than number them in their name.

Agree and done. Thanks!

Co-authored-by: Judy Racusin <[email protected]>

shb46 · 2024-11-19T19:19:55Z

@Vidushi-GitHub: Are we ready to close this PR? Who is going to close it?

Vidushi-GitHub · 2024-11-19T19:30:57Z

@Vidushi-GitHub: Are we ready to close this PR? Who is going to close it?

No, we need approvals for the latest feedbacks. I will remind Judy.

shb46 requested a review from Vidushi-GitHub September 25, 2024 14:56

shb46 marked this pull request as draft September 25, 2024 14:56

shb46 self-assigned this Sep 25, 2024

shb46 requested a review from jracusin October 1, 2024 00:56

Vidushi-GitHub reviewed Oct 2, 2024

View reviewed changes

gcn/notices/fermi/gbm/FermiGBMAlert.schema.json Outdated Show resolved Hide resolved

Vidushi-GitHub reviewed Oct 2, 2024

View reviewed changes

shb46 requested a review from Vidushi-GitHub October 2, 2024 14:16

dakota002 mentioned this pull request Oct 9, 2024

Fermi-GBM JSON Notices nasa-gcn/gcn.nasa.gov#2605

Open

5 tasks

shb46 force-pushed the fermi branch from 0a8ef0e to 890537f Compare October 15, 2024 17:32

Fermi gbm

dc47b17

dakota002 force-pushed the fermi branch from 890537f to dc47b17 Compare October 15, 2024 19:35

dakota002 marked this pull request as ready for review October 15, 2024 19:42

shb46 added 2 commits October 17, 2024 12:05

commit

42ec420

Corrections to examples

96bf975

shb46 force-pushed the fermi branch from 53c2cd0 to 96bf975 Compare October 17, 2024 17:16

Vidushi-GitHub reviewed Nov 5, 2024

View reviewed changes

gcn/notices/fermi/gbm/strict/alert.2.example.json Outdated Show resolved Hide resolved

shb46 added 5 commits November 5, 2024 20:16

Issue nasa-gcn#199: hitl -> human_in_th_loop

c3bb0a8

Issue nasa-gcn#199: Elaborate on "detectors_status" description.

5d7ca02

Issue nasa-gcn#199: Typo in the rate_duration description

c5576ab

Issue nasa-gcn#199: Specify explicitly "rate_energy_range", as the nu…

6bf5331

…mber with which the units go together."

Issue nasa-gcn#199: "real-life" -> "streamed/real".

cb904dd

Vidushi-GitHub reviewed Nov 6, 2024

View reviewed changes

Issue nasa-gcn#199: Renamed fields in examples to reflect schema

07f1cbc

Vidushi-GitHub approved these changes Nov 8, 2024

View reviewed changes

Expanded trigger_algorithm_number description

36eeb63

jracusin reviewed Nov 16, 2024

View reviewed changes

gcn/notices/fermi/gbm/strict/alert.schema.json Outdated Show resolved Hide resolved

jracusin reviewed Nov 16, 2024

View reviewed changes

shb46 added 4 commits November 16, 2024 11:28

Added const to notice_type

98a5149

Shorter human_in_the_loop description and a typo

b16182c

Shortened description and added citation to trigger_algorithm_number

30ea0d0

More descriptive example file names

c41b35d

shb46 mentioned this pull request Nov 17, 2024

Added JSON section to the Fermi mission page nasa-gcn/gcn.nasa.gov#2650

Merged

Update gcn/notices/fermi/gbm/strict/alert.schema.json

72314e2

Co-authored-by: Judy Racusin <[email protected]>

Vidushi-GitHub requested review from jracusin and lpsinger November 19, 2024 19:28

shb46 added 2 commits November 26, 2024 10:27

rm strict directory

7bcb87a

Moved strict examples here

802581c

Fermi #199

Are you sure you want to change the base?

Fermi #199

Conversation

shb46 commented Sep 20, 2024

Vidushi-GitHub Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Vidushi-GitHub commented Oct 2, 2024

Vidushi-GitHub commented Oct 2, 2024

shb46 commented Oct 2, 2024 • edited Loading

Vidushi-GitHub commented Oct 10, 2024

Vidushi-GitHub commented Oct 10, 2024

Vidushi-GitHub commented Oct 10, 2024

shb46 commented Oct 10, 2024

shb46 commented Oct 10, 2024

shb46 commented Oct 11, 2024

dakota002 commented Oct 11, 2024

shb46 commented Oct 11, 2024

dakota002 commented Oct 11, 2024

shb46 commented Oct 11, 2024

dakota002 commented Oct 11, 2024

shb46 commented Oct 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vidushi-GitHub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vidushi-GitHub Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shb46 Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shb46 commented Nov 19, 2024

Vidushi-GitHub commented Nov 19, 2024 • edited Loading

Vidushi-GitHub Oct 2, 2024 •

edited

Loading

shb46 commented Oct 2, 2024 •

edited

Loading

Vidushi-GitHub Nov 26, 2024 •

edited

Loading

shb46 Nov 26, 2024 •

edited

Loading

Vidushi-GitHub commented Nov 19, 2024 •

edited

Loading