- ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE. Jeff Waldstreicher Scientific Services Division Eastern Region Northeast Regional Operational Workshop (NROW) November 4-5, 2003 Albany, NY. ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS. - PowerPoint PPT Presentation
ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCEJeff WaldstreicherScientific Services Division Eastern RegionNortheast Regional Operational Workshop (NROW)November 4-5, 2003Albany, NYASSESSING THE IMPACTS OF COLLABORATIVE PROJECTSIMPACT ON VERIFICATION SCORESPerformance MetricsCASE STUDY ANALYSISEvent VerificationSubjective Evaluation of Impact on Forecast ProcessAFDsEvent Reviews and Impact ReportsCOMPLICATIONSMany Factors Influence Performance MetricsFactors Not IndependentDifficult to tie performance changes to a specific factorCannot Analyze Null Case If forecasters have knowledge or data, cannot directly answer What if they did not have the knowledge?FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES Infusion of New Technologies Hardware (New systems or Processors) Software (New algorithms or Models)Applied Research and DevelopmentNational (Research Laboratory)Local/RegionalIndependentCollaborativeChanges to Operational ProceduresImplementation of Best PracticesFACTORS THAT INFLUENCE WARNING VERIFICATION SCORESClimate VariabilityVariations in frequency and type of eventsExternal Outreach and EducationIncluding development of spotter networksPersonnel/Staffing IssuesIncluding forecaster experience levelsTrainingAll other factors are also tied to training issues HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?Can be global (national), regional, or localTechnology usually nationalClimate Variability typically regionalStaffing usually local Impact can be long or short termTechnology infusion typically has long term impactClimate Variability is usually short termFactors are often inter-relatedNew technology or staffing changes require training Applied R&D often yields/suggests changes to operationsCOMET PROJECTS IN EASTERN REGIONDifferent types of projectsCooperative (2-3 year ~$35K/yr)19 ER Cooperative Projects funded since 1991 Partners (1 year ~$9K/yr)39 ER Partners Projects funded since 199120 ER WFOs and 3 RFC have participated90 Offices Nationwide21 Universities have participated in ER projects70+ Universities NationwideEVALUATING COMET PROJECTSExamined COMET Cooperative and Partners Projects in Eastern RegionProjects completed between 1995 and mid-2001Projects specifically addressing warning programsTornadoSevere ThunderstormsFlash FloodingWinter StormsStudy was designed to minimize as much as possible the impact of the factors previously discussed.METHODOLOGY3-year running verification scores usedMinimize impact of short-term factors such as variability of eventsCompared 3 years before project to 3 years following project1996-mid 2001 period helped ensure 3-years of post-88D data in before scores and a full 3 years of after scores3-year expected improvements calculated based on long term trend of ER-wide scoresCompared rate of improvement for WFO involved in collaborative project to ER-wide improvementUse of ER scores as a baseline minimizes the impact of national/region-wide factors such as AWIPS and radar system improvementsVery difficult to evaluate the impact of project results beyond the primary WFOINDIVIDUAL PROJECT COMPARISONCompare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time periodIMPACT OF LONG TERM COLLABORATIVE ACTIVITIESWFO RAH and NCSUContinuous collaborative projects since January 1991COMET Projects 1991-20003 Cooperative Projects3 Partners Projects1 Graduate FellowshipCSTAR Projects2000-20032003-2006Unique laboratory to examine the impact of long-term collaborative activities IMPACT OF LONG TERM COLLABORATIVE ACTIVITIESWFO ALY and U-AlbanyContinuous since February 1995Numerous COMET Projects3 Cooperative Projects7 Partners ProjectsSeveral additional projects between UA and NCEP CSTAR PROJECT 2001-2004SUMMARYVerification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance There are indications that certain performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.