* The 1st Earthquake Forecast Testing Experiment for Japan: Call for Forecast Models, Testing Regions, and Forecast Evaluation Methods Research group "Earthquake Forecast System based on Seismicity of Japan (EFSSJ)" in collaboration with D. Schorlemmer (SCEC), F. Euchner (ETH), and CSEP Working Group K. Z. Nanjo, N. Hirata, and H. Tsuruoka (Earthquake Research Institute, University of Tokyo) are responsible for the wording of this web page. Last modified on May 2009 [Japanese version webpage|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/wiki.jp/wiki.cgi?page=ExtendedSummary_jp] - ''Abstract'' Research group "Earthquake Forecast System based on Seismicity of Japan (EFSSJ) (tentative title)" (the group's secretariat is located at Earthquake Research Institute (ERI), University of Tokyo) invites statistics- and physics-based models to statistically forecast seismicity in and around Japan based on geophysical data (for example, past and current seismicity data and crustal dynamics data from Global Positioning Systems or GPS) in order to conduct a comparative test study in determining which of these collected models are best for their forecast accuracy function. Main purpose is to evaluate the forecast performance of such submitted models and to better understand the physics and statistics of earthquake occurrence. The EFSSJ expects a researcher that is interested in submitting a model applicable only to a limited region. To accept this type of model, such region is also invited as a testing region in the objective testing framework. Further, the EFSSJ invites methodologies that can be used to measure earthquake forecast accuracy, for the purpose of examining the validation of the suitability of these collected methodologies. This is the 1st earthquake forecast testing experiment. The 2nd trail of this type of experiment will be scheduled next year. -''1. Introduction'' Japan is one of the most earthquake-prone countries in the world. A sound scientific development of earthquake prediction research for Japan is strongly urged upon emerging social demands for protecting people's lives and properties. Therefore the implementation of reliable earthquake forecasts plays an important role on responding to these demands. To enhance this reliability a scientifically rigorous study needs to be carried out to test the usability of proposed earthquake prediction (forecast) methods and to clear the conditions under which these methods are useful. Existing Japanese research projects aiming at the realization of scientific earthquake prediction have so far focused on better understanding of the mechanism of earthquake occurrence and the development of forecast simulation technologies based on physical modeling of earthquakes. However, the building of a framework for rigorous testing of earthquake forecasts and the creation of testable forecast models have never been central problems to solve for the earthquake related research. This is now a key challenge under the national "[Observation and Research Program for Prediction of Earthquake and Volcanic Eruption|http://www.mext.go.jp/b_menu/houdou/20/07/08071504/002.htm] (2009-2013)". Together with other subprograms associated with forecast simulation analysis of crustal dynamics, the research program on "Earthquake Forecast System based on Seismicity of Japan (EFSSJ) (tentative name)" will tackle the task. Our scientific experiment of testing earthquake forecasts plays an important role on a part of the latter subprogram (EFSSJ). The "[Collaboratory for the Study of Earthquake Predictability (CSEP)|http://www.cseptesting.org]" is a global project of earthquake predictability research and has started in 2006 ['''Jordan''', 2006]. This is a successor of the California project "[Regional Likelihood Models (RELM)|http://relm.cseptesting.org]" [Special issue in '''Seismol. Res. Lett., 78(1)''', 2007] that implemented an earthquake forecast testing study with several forecast classes. The primary purpose of the CSEP is to develop a virtual, distributed laboratory-a collaboratory-that can support a wide range of scientifically objective and transparent prediction experiments in multiple regional or global natural laboratories. Final goal is to look for the intrinsic predictability of the earthquake rupture process through this development. Currently the CSEP Testing Centers are operating for California, Europe, New Zealand and other regions. ERI jointed the CSEP, and prepared the Japanese Testing Center in summer 2008 with help from Southern California Earthquake Center (SCEC) and Swiss Federal Institute of Technology (or Eidgenoessische Technische Hochschule: ETH). In this preparation process, a computer system same as that used at SCEC and ETH for the CSEP data processing was installed at ERI. Using this computer system, we started conducting the experimental study for Japan ['''Tsuruoka et al.''', 2008]. The testing region is given in Fig. 1 ([pdf|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.1.pdf], [jpg|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.1.jpg]). Currently a set of three 1-year smoothed-seismicity models based on different earthquake-generation hypotheses is under test: Triple-S, JALM, and RI models (Left, middle, and right panels, respectively, in Fig. 2 ([pdf|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.2.pdf], [jpg|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.2.jpg])). Our testing approach is a prospective way that the performance for registered models is assessed through forecasting truly future earthquakes. This prospective performance assessment makes the tests more rigorous and objective because the forecast results are unknown when the scientific prediction experiments are started in operation. The three models are evaluated against the authoritative observed data maintained by Japan Meteorological Agency (JMA) with the official suite of tests used in the CSEP experiments. Currently 3 evaluation tests are used in the Testing Center at ERI: N (number of events) Test, L (log likelihood) Test, and R (log likelihood ratio) Test ['''Schorlemmer et al.''', 2007]. Additional tests expect to be integrated in near future. Using the Testing Center, we formally start conducting the comparative forecast experimental research in 2009 for Japan. For this research, we invite earthquake forecast models. The experiment objectively evaluates the performance of earthquake forecast models. This objective approach seeks to provide the progress of sophisticating the registered physics- and statistical-based models, which will contribute to the forecast accuracy improvement of these models. Testing classes are 1-day, 1-year, and 5-year forecasts. The Testing Center also expects a researcher that is interested in submitting a model applicable only to a limited region. To accept this type of model, such region is invited as a testing region. To make an international contribution to the field of earthquake predictability research, we also invite forecast evaluation methods that are different from the commonly accepted official suite of tests used in the earthquake predictability research field. -''2. The Rules of the Game'' Our approach is based on the CSEP testing centers that are modeled after the RELM: Refer the [RELM special issue|http://relm.cseptesting.org] ['''Seismol. Res. Lett., 78(1)''', 2007] and the [CSEP website|http://www.cseptesting.org]. A summary of the game's rules specific to Japan is given below. --2.1. Participants and release of test results ---2.1.1. Modelers: Researchers who submit their earthquake forecast models: each model provides with a prospective earthquake forecast. Models cannot be withdrawn from the test without agreement of the Testing Center. ---2.1.2. Testing Center: The facility with the CSEP infrastructure at ERI. Each earthquake forecast model is submitted to this Center and is evaluated independently from the modeler. All forecast results are stored: therefore additional prospective tests can be integrated at later stages without compromising integrity of prospective testing experiment. ---2.1.3. Release of test results: The information on test results is closed to public access. But password-protected web pages in the website (7) will be used to allow a participant community to access this information. A summary of test results with a brief explanation of submitted forecast models will be considered for peer-reviewed publication. --2.2. Testing regions ---2.2.1. The region that covers Japan for the depth range from 0 to 30 km shown in Fig. 1 ([pdf|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.1.pdf], [jpg|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.1.jpg]), with a predefined spatial grid with 0.1 spacing. Download here for the list of grid nodes, each at which the forecast rates for magnitude bins in the range 5.0<=M<=9.0 (0.1 magnitude unit steps) must be computed. ---2.2.2. A region that is not equal to the region in 2.2.1. To accept a model applicable only to a limited region, such region with a spatial grid is predefined for an experimental test, based on a proposal from the corresponding modeler. --2.3. Testing classes ---2.3.1. 1-day forecast: Forecast models must define a seismicity rate for each magnitude bin in the range 5.0<=M<=9.0 (0.1 magnitude unit steps) for consecutive 1-day periods starting at midnight in Japanese Standard Time. Models must be implemented at the Testing Center as code that can independently and automatically compute the forecast rates, based on predefined authoritative input data. The output containing forecast rates must follow the ForecastML format (template can be downloaded here). Forecast codes need to be installed and function-checked in collaboration with Testing Center's staffs. ---2.3.2. 1-year forecast: Forecast models must define a seismicity rate (number of earthquakes in 1-year period) for each magnitude bin in the range 5.0<=M<=9.0 (0.1 magnitude unit steps). Similarly to the 1-day forecast class, a code that can independently and automatically compute the forecast rates is preferably received by the Testing Center: the Center's staffs and modelers need to work on code installation and function-check. In case that a code cannot be made open source, the Testing Center can accept a numerical table containing forecast rates for the corresponding model in the ForecastML format whose template can be downloaded here. ---2.3.3. 5-year forecast: This is the same as the 1-year forecast class, except for the time window length. --2.4. Future earthquakes that will be forecast: The official bulletin for future earthquakes is the revised JMA bulletin. No declustering is applied to the JMA catalog. In other words, models will be evaluated against observed earthquake data in the JMA nondeclustered catalog using the forecast evaluation methods described below (2.5). Forecast is a seismicity rate (number of earthquakes in a predefined time window) for each magnitude bin in the range 5.0<=M<=9.0 (0.1 magnitude unit steps) at each predefined grid node within a predefined testing region. --2.5. Forecast evaluation methods ---2.5.1. In spite of the testing classes (2.3), the official suite of tests used in the CSEP is used. Currently the Testing Center at ERI uses the 3 evaluation tests (N-, L-, and R-Tests) ['''Schorlemmer et al.''', 2007]. ---2.5.2. To make an international contribution to the field of earthquake predictability research, we invite forecast evaluation methods that are different from the official suite of tests used in the CSEP Testing Centers. -''3. Earthquake Forecast Model Proposal'' Each model must forecast seismicity rates for locations given in the list (3.1) and output a numerical table containing forecast rates in the format according to the ForecastML (3.2). --3.1. Grid node list for the testing region: longitude and latitude pair, [Download|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/japan.forecast.nodes.dat] --3.2. Forecast template (ForecastML), [Download|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/csep.ri.xml] If the JMA catalog and other data such as GPS data are needed for model optimization, contact the Testing Center. Modelers must download the application form ([English|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/ForecastModelProposal_en.doc], [Japanese|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/ForecastModelProposal_jp.doc]) to give notice of forecast model proposal, fill in the form, and send it back to the Testing Center before the deadline date (6.1). After this notice, each modeler will collaborate with Testing Center's staffs in order to work on installation and function-check of his or her model code. A modeler who cannot be made his or her code as an open source for the testing classes of 1- year and 5-year forecasts (2.3.2 and 2.3.3) must contact before the deadline date (6.1) so that a numerical table formatted according to (3.2) will be accepted. -''4. Testing Region Proposal'' Download the application form ([English|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestRegionProposal_en.doc], [Japanese|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestRegionProposal_jp.doc]) to give notice of testing region proposal, fill in the form, and send it back to the Testing Center before the deadline date (6.2). After this notice, each will work with the Testing Center on a case-by-case basis to find a solution that fits the needs of both sides. -''5. Forecast Evaluation Method Proposal'' Download the application form ([English|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestingMethodProposal_en.doc], [Japanese|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestingMethodProposal_jp.doc]) to give notice of evaluation method proposal, fill in the form, and send it back to the Testing Center before the deadline date (6.3). After this notice, each corresponding researcher and the Testing Center will work on a case-by-case basis on finding a solution that fits the needs of both sides. -''6. Deadlines and application forms for the three notices to the Testing Center for the 1st experiment'' Here is a set of the application form and deadline date for the three proposals above (3-5) for the 1st earthquake forecast testing experiment. After receiving the form from each applicant, Testing Center's staff will contact the applicant. --6.1. Earthquake forecast model: 21 May 2009, Download ([English|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/ForecastModelProposal_en.doc], [Japanese|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/ForecastModelProposal_jp.doc]) --6.2. Testing region: 11 May 2009, Download ([English|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestRegionProposal_en.doc], [Japanese|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestRegionProposal_jp.doc]) --6.3. Forecast evaluation method: 11 May 2009, Download ([English|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestingMethodProposal_en.doc], [Japanese|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/TestingMethodProposal_jp.doc]) -''7. Contact information'' The secretariat (N. Hirata, H. Tsuruoka, and K. Z. Nanjo) of the research group "Earthquake Forecast System based on Seismicity of Japan (EFSSJ)" is responsible for the Testing Center at Earthquake Research Institute, University of Tokyo, 1-1-1 Yayoi, Bunkyo-ku, 113-0032 Tokyo, Japan. --E-mail: ZISINyosoku-submit@eri.u-tokyo.ac.jp --Website: http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku -''8. Acknowledgement'' We thank D. Schorlemmer (SCEC), F. Euchner (ETH), and CSEP Working Group for installing an earlier version of the CSEP infrastructure operated at ERI. -''9. References'' --Jordan, T. H. (2006), Earthquake predictability, brick by brick, '''Seismol. Res. Lett.''', 77, 3-6. --Japan Meteorological Agency (2002), Extension of the seismological catalogue of the Japan Meteorological Agency (August, 1923-December, 1925), '''Jishin Yochi Renrakukai Kaiho''', 67, 491-498 (in Japanese). --Schorlemmer, D. et al. (2007), Earthquake likelihood model testing, '''Seismol. Res. Lett.''', 78(1), 17-29. --Tsuruoka, H. et al. (2008), The Collaboratory for the Study of Earthquake Predictability: Establishing a testing center in Japan, In '''The 7th General Assembly of Asian Seismological Commission and Seismological Society of Japan, 2008 Fall Meeting''' (24-27 Nov. 2008, Tsukuba, Japan), Program and abstract, p. C41-09. -''10. Figures'' ''Fig. 1.'' The testing region that is currently used for the 1-year forecast class and will be used for 2.2.1. The grid spacing is 0.1 degree. From '''Tsuruoka et al.''' [2008]. Download ([pdf|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.1.pdf], [jpg|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.1.jpg]) ''Fig. 2.'' The 1-year smoothed-seismicity models running currently in the Testing Center: Triple-S (Left), JALM (Middle), and RI (Right). The color bar in each panel shows expectations for the annual number of earthquakes. Note that the scale varies from one panel to the other. Download ([pdf|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.2.pdf], [jpg|http://wwweic.eri.u-tokyo.ac.jp/ZISINyosoku/downloads/TestCenterV1/Fig.2.jpg])