Subject: Administrative: Status of GOES-16 GLM L2 data, Issued: --001a114b1a305342900563733b68 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Topic: Status of GOES-16 GLM L2 Data Date/Time Issued: January 23, 2018 1530 UTC Product(s) or Data Impacted: GOES-16 GLM L2 Data Date/Time of Initial Impact: January 23, 2018 at 1530 UTC Date/Time of Expected End: N/A Length of Event: N/A Details/Specifics of Change: The GOES-R Peer Stakeholder - Product Validation Review for GLM L2 data Provisional Maturity was held on January 19, 2018. As a result of this review, NOAA has confirmed that the GLM L2 data are at Provisional Validation Maturity as of January 19, 2018. The GLM L2 product consists of geo-located and time-stamped events, groups, and flashes, with associated calibrated optical amplitudes (in units of Joules). Provisional maturity means: - Validation activities are ongoing and the general research community is now encouraged to participate. - Severe algorithm anomalies are identified and under analysis. Solutions to anomalies are in development and testing. - Incremental product improvements may still be occurring. - Product performance has been demonstrated through analysis of a small number of independent measurements from select locations, periods, and associated ground truth and field campaign efforts. - Product analysis is sufficient to establish product performance relative to expectations. - Documentation of product performance exists that includes recommended remediation strategies for all anomalies and weaknesses. Any algorithm changes associated with severe anomalies have been documented, implement= ed, and tested. Provisional users bear all responsibility for inspecting the data prior to use and for the manner in which the data are utilized. Persons desiring to use the GOES-16 GLM Provisional-maturity L2 products for any reason, including but not limited to scientific and technical investigations, are encouraged to consult the GLM Calibration/Validation science team members for feasibility of the planned applications. Commentary on important GLM data characteristics and known issues being resolved include: 1. Flash Detection: GLM flash detection efficiency (FDE) over the field-of-view and over a 24 hr period appears to exceed 70%. However, si= nce the FDE associated with reference data are below 100%, it is difficult t= o exactly determine the GLM FDE. In addition, the FDE is substantially smaller in anomalous (i.e. inverted polarity) storms. Finally, erroneous GLM flash splitting occurs (but is rare and typically only happens with large, long duration flashes) and a solution is presently under analysis= . 1. False Events: Significant reductions in false events have been achieved in Provisional relative to Beta. High energy radiation particles, aka "radiation dots", have been reduced by removing Single Group Flashes (SGFs), but future adjustments to this filter will be tested so as to mitigate removal of natural flashes. False events due to the data handli= ng issue that produced =E2=80=9Cduplicative dots=E2=80=9D has been fully re= solved. False events due to blooming, created from glint and/or solar intrusion, are still present but a blooming filter and improvements to glint processing are planned. 1. Position Errors: Significant reduction in lightning position errors have been achieved in Provisional relative to Beta. There is now subpixel accuracy over most of the GLM domain, but GLM positions near the edge of the field-of-view must be shifted outward to match reference data. Overa= ll, the distribution of the errors between GLM optical group locations and t= he reference data peaks at about 4 km; if one considers just group location= s near the edge of the field-of-view the distribution peaks at about 11 km= . Mitigation of parallax errors that arise from inaccurate cloud-top height estimation, and ways to combat location errors that are due to side-cloud illumination are being tested. Mitigation of any INR inaccuracies due to diurnal variations will also be examined. 1. Timestamp: 1. TOF: The timestamp on events, groups, and flashes still do not properly account for the Time-Of-Flight (TOF) of the photons from cloud-top to sensor. Until a formal fix is applied, we recommend that= the user apply their own TOF correction of about 125 ms [i.e., the actual= TOF correction is estimated to range from 119 ms (nadir) to 131 ms (edge = of FOV)]. With this adjustment, the distribution of the errors between G= LM group times and reference data time peaks at 0.7 ms (K. Virts analysi= s). 2. The sequence in a L2 data file of event times (as well as group time, flash start time, and flash end time) are not in time order. 1. Family Links: There exists =E2=80=9Corphan=E2=80=9D and =E2=80=9Cchildle= ss=E2=80=9D events and groups. For example, a =E2=80=9Cchildless=E2=80=9D group is a group with no even= ts; an =E2=80=9Corphan=E2=80=9D event is an event with no parent group. 1. Areas: Small discrepancies in group and flash area values are being examined. 1. Unsigned Integer Read: This is an issue that affects multiple instruments on GOES-16, and a pilot fix is being worked. The classic mod= el for NetCDF does not support unsigned integers larger than 8 bits. Many o= f the variables in GOES-16 data files are unsigned integers that are eithe= r 16-bits or 32-bits. So, until a fix is achieved, we recommend using the following process to convert: 1. Retrieve the variable data (using low level routines). 2. If there is an attribute =E2=80=9C_Unsigned=E2=80=9D then cast the va= riable data to unsigned. 3. This step must be completed before applying scale_factor and add_offset values to convert from scaled integer to science units. Fo= r example, when reading the NetCDF files, one has to MANUALLY read in t= he event lat/lon as an unsigned integer (using low level routines), and = then manually take care of the scale and offset. 1. Data Quality Product: This product is not yet available but is in planning. 1. Data Recommended: The Calibration Working Group (CWG) recommends that data on or after January 11, 2018 is best to use since it follows the important algorithm patches (DO.06.02.00 in November, PR.06.05.00 in ear= ly January), INR averaging, satellite drift, and settling the INR averaging= at the East slot. Contact Information for Further Information: Kathryn Mozer at Kathryn.Mozer@noaa.gov. Web Site(s) for applicable information: See http://www.ssd.noaa.gov/PS/SATS/messages.html for this and other satellite related messages. See https://www.goes-r.gov/spacesegment/glm.html for GLM specific information. This message was sent from ESPC.Notification@noaa.gov. You have been sent this and other notifications because you have opted in to receive it. If for any reason, you wish to unsubscribe, please contact ESPC Help Desk at ESPCOperations@noaa.gov or (301) 817-3880. Please note: it may take up to two business days to process your unsubscribe request. --001a114b1a305342900563733b68 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Topic: Status of GOES-16 G= LM L2 Data

Date/Time I= ssued: January 23, 2018 1530 UTC

Product(s) or Data Impacted: GOES-16 GLM L2 Data

Da= te/Time of Initial Impact: January 23, 2018 at 1530= UTC

Date/Time of Expected End= : =C2=A0N/A

Length of Event: =C2=A0N/A


Details/Specifics of Change: The GOES-R Peer Stakeholder - Product Validation Review for GLM L2 = data Provisional Maturity was held on January 19, 2018. As a result of this= review, NOAA has confirmed that the GLM L2 data are at Provisional Validat= ion Maturity as of January 19, 2018. =C2=A0

=C2=A0

The GLM L2 product consists of geo-located and = time-stamped events, groups, and = flashes, with associated calibrated optical amplitu= des (in units of Joules).

Pr= ovisional maturity means:

  • Validation activities ar= e ongoing and the general research community is now encouraged to participa= te.

  • Severe algor= ithm anomalies are identified and under analysis. Solutions to anomalies ar= e in development and testing.

  • Incremental product improvements may still be occurring.

  • Product performance = has been demonstrated through analysis of a small number of independent mea= surements from select locations, periods, and associated ground truth and f= ield campaign efforts.

  • Product analysis is sufficient to establish product performance relat= ive to expectations.

  • Documentation of product performance exists that includes recommended r= emediation strategies for all anomalies and weaknesses. Any algorithm chang= es associated with severe anomalies have been documented, implemented, and = tested.


Provis= ional users bear all responsibility for inspecting the data prior to use an= d for the manner in which the data are utilized. Persons desiring to use th= e GOES-16 GLM Provisional-maturity L2 products for any reason, including bu= t not limited to scientific and technical investigations, are encouraged to= consult the GLM Calibration/Validation science team members for feasibilit= y of the planned applications. Commentary on important GLM data characteris= tics and known issues being resolved include:

  1. <= p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;te= xt-align:justify">Flash Detection: GL= M flash detection efficiency (FDE) over the field-of-view and over a 24 hr = period appears to exceed 70%. However, since the FDE associated with refere= nce data are below 100%, it is difficult to exactly determine the GLM FDE. = In addition, the FDE is substantially smaller in anomalous (i.e. in= verted polarity) storms= . Finally, erroneous GLM flash splitting occurs (but is rare and typically only happens wi= th large, long duration flashes) and a solution is presently under analysis= .


  1. = False Events:<= /span> Significant reductions = in false events have been achieved in Provisional relative to Beta. High en= ergy radiation particles, aka "radiation dots", have been reduced= by removing Single Group Flashes (SGFs), but future adjustments to this fi= lter will be tested so as to mitigate removal of natural flashes. False eve= nts due to the data handling issue that produced =E2=80=9Cduplicative dots= =E2=80=9D has been fully resolved. False events due to blooming, created from glint and/= or solar intrusion, are still present but a blooming filter and improvement= s to glint processing are planned.


  1. Position Errors: Significant reduction in lightning position errors have been a= chieved in Provisional relative to Beta. There is now subpixel accuracy ove= r most of the GLM domain, but GLM positions near the edge of the field-of-v= iew must be shifted outward to match reference data. Overall, the distribut= ion of the errors between GLM optical group locations and the reference dat= a peaks at about 4 km; if one considers just group locations near the edge = of the field-of-view the distribution peaks at about 11 km. Mitigation of <= /span>parallax errors = that arise from inaccurate cloud-top height estimation, and ways to combat = location errors that are due to side-cloud illumination are being tested.= =C2=A0 Mitigation of any INR inaccuracies due to diurnal variations= will also be examined.=


  1. <= span style=3D"font-weight:700;font-variant-numeric:normal;font-variant-east= -asian:normal;vertical-align:baseline;white-space:pre-wrap">Timestamp:

    1. TOF: The timest= amp on events, groups, and flashes still do not properly account for the Time-Of-Flight (TOF) of the photons from cloud-top t= o sensor. Until a formal fix is applied, we recommend that the user apply t= heir own TOF correction of about 125 ms [i.e., the actual TOF correction is= estimated to range from 119 ms (nadir) to 131 ms (edge of FOV)]. With this= adjustment, the distribution of the errors between GLM group times and ref= erence data time peaks at 0.7 ms (K. Virts analysis).

    2. The s= equence in a L2 data file of event times (as well as group time, flash star= t time, and flash end time) are not in time ord= er.=


  1. Family Links: There exists =E2=80=9Corphan=E2=80= =9D and =E2=80=9Cchildless=E2=80=9D events and groups. For example, a =E2= =80=9Cchildless=E2=80=9D group is a group with no events; an =E2=80=9Corpha= n=E2=80=9D event is an event with no parent group.


=

  1. Areas: Small discrepancies in group and flash area values are b= eing examined.

  1. Unsigned Integer Read: This is an issue that affects multiple instruments on GOES-1= 6, and a pilot fix is being worked. The classic model for NetCDF does not s= upport unsigned integers larger than 8 bits. Many of the variables in GOES-= 16 data files are unsigned integers that are either 16-bits or 32-bits. So,= until a fix is achieved, we recommend using the following process to conve= rt:

    1. Retrieve the variable data (using low level routines= ). =C2=A0

    2. If there is an attri= bute =E2=80=9C_Unsigned=E2=80=9D then cast the variable data to unsigned.

    3. This step must be completed bef= ore applying scale_factor and add_offset values to convert from scaled inte= ger to science units. For example, when reading the NetCDF files, one has t= o MANUALLY read in the event lat/lon as an unsigned integer (using low leve= l routines), and then manually take care of the scale and offset.


  1. Data Quality Product= : This product is not y= et available but is in planning.


  1. Data Recommended: The Calibration Working Group (CWG) recommends that data on or = after January 11, 2018 is best to use since it follows the important algori= thm patches (DO.06.02.00 in November, PR.06.05.00 in early January), INR av= eraging, satellite drift, and settling the INR averaging at the East slot. =


Contact Infor= mation for Further Information: Kathryn Mozer at Kathryn.Mozer@= noaa.gov. =C2=A0


Web Site(s) for applicable information: See http://www.ssd.noaa.gov/PS/SATS/messages.html for this and other satellite related messages. See https://www.g= oes-r.gov/spacesegment/glm.html for GLM specifi= c information.


This message was sent from <= /span>ESPC.Notification@noaa.gov. You = have been sent this and other notifications because you have opted in to re= ceive it. If for any reason, you wish to unsubscribe, please contact ESPC H= elp Desk at ESPCOperations@noaa.gov or (301) 817-3880. Please note: it may take up to two business days to process = your unsubscribe request.


--001a114b1a305342900563733b68--