import ocean_model_skill_assessor as omsa
from IPython.display import Code, Image

CLI demo of ocean-model-skill-assessor with known data files#

This demo runs command line interface (CLI) commands only, which is accomplished in a Jupyter notebook by prefacing commands with !. To transfer these commands to a terminal window, remove the ! but otherwise keep commands the same.

More detailed docs about running with the CLI are available.

There are three steps to follow for a set of model-data validation, which is for one variable:

  1. Make a catalog for your model output.

  2. Make a catalog for your data.

  3. Run the comparison.

These steps will save files into a user application directory cache, along with a log. A project directory can be checked on the command line with omsa proj_path --project_name PROJECT_NAME.

Make model catalog#

Set up a catalog file for your model output. The user can input necessary keyword arguments – through kwargs_open – so that xarray will be able to read in the model output. Generally it is good to use skip_entry_metadata when using the make_catalog command for model output since we are using only one model and the entry metadata is aimed at being able to compare datasets.

In the following command,

  • make_catalog is the function being run from OMSA

  • demo_local is the name of the project which will be used as the subdirectory name

  • local is the type of catalog to choose when making a catalog for the model output regardless of where the model output is stored

  • “model” is the catalog name which will be used for the file name and in the catalog itself

  • Specific kwargs to be input to the catalog command are

    • filenames which is a string describing where the model output can be found. If the model output is available through a sequence of filenames instead of a single server address, represent them with a single glob-style statement, for example, “/filepath/filenameprefix_*.nc”.

    • skip_entry_metadata use this when running make_catalog for model output

  • kwargs_open all keywords required for xr.open_dataset or xr.open_mfdataset to successfully read your model output.

# get local path for model output sample file from xroms
import xroms
url = xroms.datasets.CLOVER.fetch("ROMS_example_full_grid.nc")

!omsa make_catalog --project_name demo_local --catalog_type local --catalog_name model --kwargs filenames=$url skip_entry_metadata=True

[2023-11-27 22:04:51,283] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:430}
INFO - Catalog saved to /home/docs/.cache/ocean-model-skill-assessor/demo_local/model.yaml with 1 entries.

Make data catalog#

Set up a catalog of the datasets with which you want to compare your model output. In this example, we use only known data file locations to create our catalog.

In this step, we use the same project_name as in the previous step so as to put the resulting catalog file in the same subdirectory, we create a catalog of type “local” since we have known data locations, we call this catalog file “local”, input the filenames as a list in quotes (this specific syntax is necessary for inputting a list in through the command line interface), and we input any keyword arguments necessary for reading the datasets.

In the following command:

  • make_catalog is the function being run from OMSA

  • demo_local is the name of the project which will be used as the subdirectory name

  • local is the type of catalog to choose when making a catalog for the known data files

  • “local” is the catalog name which will be used for the file name and in the catalog itself

  • Specific kwargs to be input to the catalog command are

    • filenames which is a string or a list of strings pointing to where the data files can be found. If you are using a list, the syntax for the command line interface is filenames="[file1,file2]".

  • kwargs_open all keywords required for xr.open_dataset or xr.open_mfdataset or pandas.open_csv, or whatever method will ultimately be used to successfully read your model output. These must be applicable to all datasets represted by filenames. If they are not, run this command multiple times, one for each set of filenames and kwargs_open that match.

!omsa make_catalog --project_name demo_local --catalog_type local --catalog_name local --kwargs filenames="[https://erddap.sensors.axds.co/erddap/tabledap/gov_ornl_cdiac_coastalms_88w_30n.csvp?time%2Clatitude%2Clongitude%2Cz%2Csea_water_temperature&time%3E=2009-11-19T012%3A00%3A00Z&time%3C=2009-11-19T16%3A00%3A00Z]" --metadata featuretype=timeSeries maptype=point

[2023-11-27 22:05:01,869] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:216}
WARNING - Dataset gov_ornl_cdiac_coastalms_88w_30n had a timezone UTC which is being removed. Make sure the timezone matches the model output.

[2023-11-27 22:05:01,883] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:430}
INFO - Catalog saved to /home/docs/.cache/ocean-model-skill-assessor/demo_local/local.yaml with 1 entries.

Run comparison#

Now that the model output and dataset catalogs are prepared, we can run the comparison of the two.

In this step, we use the same project_name as the other steps so as to keep all files in the same subdirectory. We input the data catalog name under catalog_names and the model catalog name under model_name.

At this point we need to select a single variable to compare between the model and datasets, and this requires a little extra input. Because we don’t know anything about the format of any given input data file, variables will be interpreted with some flexibility in the form of a set of regular expressions. In the present case, we will compare the water temperature between the model and the datasets (the model output and datasets selected for our catalogs should contain the variable we want to compare). Several sets of regular expressions, called “vocabularies”, are available with the package to be used for this purpose, and in this case we will use one called “general” which should match many commonly-used variable names. “general” is selected under vocab_names, and the particular key from the general vocabulary that we are comparing is selected with key.

See the vocabulary here.

import cf_pandas as cfp

paths = omsa.paths.Paths()
vocab = cfp.Vocab(paths.VOCAB_PATH("general"))
vocab

{'temp': {'name': '(?i)^(?!.*(air|qc|status|atmospheric|bottom|dew)).*(temp|sst).*'}, 'salt': {'name': '(?i)^(?!.*(soil|qc|status|bottom)).*(sal|sss).*'}, 'ssh': {'name': '(?i)^(?!.*(qc|status)).*(sea_surface_height|surface_elevation|zeta).*'}, 'u': {'name': 'u$|(?i)(?=.*east)(?=.*vel)'}, 'v': {'name': 'v$|(?i)(?=.*north)(?=.*vel)'}, 'w': {'name': 'w$|(?i)(?=.*up)(?=.*vel)'}, 'water_dir': {'name': '(?i)^(?!.*(qc|status|air|wind))(?=.*dir)(?=.*water)'}, 'water_speed': {'name': '(?i)^(?!.*(qc|status|air|wind))(?=.*speed)(?=.*water)'}, 'wind_dir': {'name': '(?i)^(?!.*(qc|status|water))(?=.*dir)(?=.*wind)'}, 'wind_speed': {'name': '(?i)^(?!.*(qc|status|water))(?=.*speed)(?=.*wind)'}, 'sea_ice_u': {'name': '(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*u)|(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*x)(?=.*vel)|(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*east)(?=.*vel)'}, 'sea_ice_v': {'name': '(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*v)|(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*y)(?=.*vel)|(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*north)(?=.*vel)'}, 'sea_ice_area_fraction': {'name': '(?i)^(?!.*(qc|status))(?=.*sea)(?=.*ice)(?=.*area)(?=.*fraction)'}}

In the following command:

  • run is the function being run from OMSA

  • demo_local is the name of the project which will be used as the subdirectory name

  • catalog_names are the names of any catalogs with datasets to include in the comparison. In this case we have just one called “local”

  • model_name is the name of the model catalog we previously created

  • vocab_names are the names of the vocabularies to use for interpreting which variable to compare from the model output and datasets. If multiple are input, they are combined together. The variable nicknames need to match in the vocabularies to be interpreted together.

  • key is the nickname or alias of the variable as given in the input vocabulary

!omsa run --project_name demo_local --catalog_names local --model_name model --vocab_names general \
        --key temp \
      --kwargs_map label_with_station_name=True \
          --more_kwargs interpolate_horizontal=False check_in_boundary=False plot_map=True dd=5 alpha=20

[2023-11-27 22:05:10,950] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1803}
INFO - Input parameters: {'catalogs': ['local'], 'project_name': 'demo_local', 'key_variable': 'temp', 'model_name': 'model', 'vocabs': ['general'], 'vocab_labels': None, 'ndatasets': None, 'kwargs_map': {'label_with_station_name': True}, 'verbose': True, 'mode': 'w', 'testing': False, 'alpha': 20, 'dd': 5, 'preprocess': False, 'need_xgcm_grid': False, 'xcmocean_options': None, 'kwargs_xroms': None, 'locstream': True, 'interpolate_horizontal': False, 'horizontal_interp_code': 'delaunay', 'save_horizontal_interp_weights': True, 'want_vertical_interp': False, 'extrap': False, 'model_source_name': None, 'catalog_source_names': None, 'user_min_time': None, 'user_max_time': None, 'check_in_boundary': False, 'tidal_filtering': None, 'ts_mods': None, 'model_only': False, 'plot_map': True, 'no_Z': False, 'skip_mask': False, 'wetdry': False, 'plot_count_title': True, 'cache_dir': None, 'return_fig': False, 'override_model': False, 'override_processed': False, 'override_stats': False, 'override_plot': False, 'plot_description': None, 'kwargs_plot': None, 'skip_key_variable_check': False, 'kwargs': {}, 'paths': <ocean_model_skill_assessor.paths.Paths object at 0x7ff5ecd33af0>, 'logger': <Logger ocean_model_skill_assessor.utils (INFO)>}

[2023-11-27 22:05:10,971] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1838}
INFO - Note that there are 1 datasets to use. This might take awhile.

[2023-11-27 22:05:10,971] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1855}
INFO - Catalog <Intake catalog: local>.

[2023-11-27 22:05:10,971] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1870}
INFO - 
source name: gov_ornl_cdiac_coastalms_88w_30n (1 of 1 for catalog <Intake catalog: local>.

[2023-11-27 22:05:11,162] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1069}
INFO - 
                        User time range: NaT to NaT.
                        Model time range: 2009-11-19 12:00:00 to 2009-11-19 16:00:00.
                        Data time range: 2009-11-19 12:17:00 to 2009-11-19 15:17:00.
                        Data lon range: -88.6 to -88.6.
                        Data lat range: 30.0 to 30.0.

[2023-11-27 22:05:11,163] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1946}
INFO - running gov_ornl_cdiac_coastalms_88w_30n for key_variable(s) temp from key_variable_list ['temp']

[2023-11-27 22:05:11,714] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:878}
INFO - Processed data file name is /home/docs/.cache/ocean-model-skill-assessor/demo_local/processed/local_gov_ornl_cdiac_coastalms_88w_30n_temp_data.csv.

[2023-11-27 22:05:11,714] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:879}
INFO - Processed model file name is /home/docs/.cache/ocean-model-skill-assessor/demo_local/processed/local_gov_ornl_cdiac_coastalms_88w_30n_temp_model.nc.

[2023-11-27 22:05:11,714] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:880}
INFO - model file name is /home/docs/.cache/ocean-model-skill-assessor/demo_local/model_output/local_gov_ornl_cdiac_coastalms_88w_30n_temp.nc.

[2023-11-27 22:05:11,714] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2004}
INFO - Figure name is /home/docs/.cache/ocean-model-skill-assessor/demo_local/out/local_gov_ornl_cdiac_coastalms_88w_30n_temp.png.

[2023-11-27 22:05:11,714] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2025}
INFO - No previously processed model output and data available for gov_ornl_cdiac_coastalms_88w_30n, so setting up now.

[2023-11-27 22:05:11,717] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1363}
INFO - Finding and saving mask to cache to /home/docs/.cache/ocean-model-skill-assessor/demo_local/mask_temp.nc.

[2023-11-27 22:05:11,717] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/utils.py:570}
INFO - Retrieving mask

[2023-11-27 22:05:14,153] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1158}
INFO - Calculating numerical domain boundary.

[2023-11-27 22:05:14,157] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:969}
WARNING - Dataset gov_ornl_cdiac_coastalms_88w_30n had a timezone UTC which is being removed. Make sure the timezone matches the model output.

[2023-11-27 22:05:14,241] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:793}
WARNING - the 'vertical' key cannot be identified in dam by cf-xarray. Maybe you need to include the xgcm grid and vertical metrics for xgcm grid, but maybe your variable does not have a vertical axis.

[2023-11-27 22:05:14,249] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:667}
INFO - Will not perform vertical interpolation and will find nearest depth to 0.0.

[2023-11-27 22:05:14,249] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1452}
INFO - Selecting model output at locations to match dataset gov_ornl_cdiac_coastalms_88w_30n.

[2023-11-27 22:05:14,307] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1484}
INFO - 
    Model coordinates found are Coordinates:
    xi_rho      int64 299
    eta_rho     int64 92
    lon_rho     float64 -88.6
    lat_rho     float64 29.97
    s_rho       float64 -0.01667
    npts        int64 0
  * ocean_time  (ocean_time) datetime64[ns] 2009-11-19T12:17:00 2009-11-19T15....
    
    Output information from finding nearest neighbors to requested points are {'distances': array([0.12069942]), 'eta_rho': array([92]), 'xi_rho': array([299])}.

[2023-11-27 22:05:14,310] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1524}
INFO - Trying to drop vertical coordinates time series

[2023-11-27 22:05:14,311] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1537}
INFO - Loading model output...

[2023-11-27 22:05:14,514] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:1644}
INFO - Saving model output to file...

[2023-11-27 22:05:14,673] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2337}
INFO - model file name is /home/docs/.cache/ocean-model-skill-assessor/demo_local/model_output/local_gov_ornl_cdiac_coastalms_88w_30n_temp.nc.

[2023-11-27 22:05:14,673] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2339}
INFO - Reading model output from file.

[2023-11-27 22:05:14,801] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2362}
INFO - Calculating stats for temp.

[2023-11-27 22:05:14,814] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/conda/latest/lib/python3.9/warnings.py:109}
WARNING - /home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/conda/latest/lib/python3.9/site-packages/numpy/lib/function_base.py:2853: RuntimeWarning: invalid value encountered in divide
  c /= stddev[:, None]


[2023-11-27 22:05:14,814] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/conda/latest/lib/python3.9/warnings.py:109}
WARNING - /home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/conda/latest/lib/python3.9/site-packages/numpy/lib/function_base.py:2854: RuntimeWarning: invalid value encountered in divide
  c /= stddev[None, :]


[2023-11-27 22:05:14,818] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/conda/latest/lib/python3.9/warnings.py:109}
WARNING - /home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/stats.py:100: RuntimeWarning: divide by zero encountered in double_scalars
  return float(1 - ((obs - model) ** 2).sum() / ((obs - obs_model) ** 2).sum())

[2023-11-27 22:05:14,823] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2380}
INFO - Saved stats file.

[2023-11-27 22:05:16,106] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2437}
INFO - Made plot for gov_ornl_cdiac_coastalms_88w_30n
.

[2023-11-27 22:05:39,261] {/home/docs/checkouts/readthedocs.org/user_builds/ocean-model-skill-assessor/checkouts/latest/ocean_model_skill_assessor/main.py:2451}
INFO - Finished analysis. Find plots, stats summaries, and log in /home/docs/.cache/ocean-model-skill-assessor/demo_local.

Look at results#

Now we can look at the results from our comparison! You can find the location of the resultant files printed at the end of the run command output above. Or you can find the path to the project directory while in Python with:

paths = omsa.paths.Paths("demo_local")
paths.PROJ_DIR

PosixPath('/home/docs/.cache/ocean-model-skill-assessor/demo_local')

Or you can use a command:

!omsa proj_path --project_name demo_local

/home/docs/.cache/ocean-model-skill-assessor/demo_local

Here we know the names of the files so show them inline.

First we see a map of the area around the Mississippi river delta, along with a red line outlining the approximate domain of the numerical model, and 1 black dot indicating 1 data location, marked with a the station name.

Image(paths.OUT_DIR / "map.png")
_images/d77d15d98a357b77f33de4d2e14d8cc0b2cf88bec22f879b83c0b24864a35d28.png

Here we see a time series comparison for station “gov_ornl_cdiac_coastalms_88w_30n”. It shows in black the temperature values from the data and in red the comparable values from the model. The comparison time range is November 19, 2009 from 12 to 15:30. The lines are not similar because the data is actually missing during this time period. Statistical comparisons are also available in the title text.

Image(paths.OUT_DIR / "local_gov_ornl_cdiac_coastalms_88w_30n_temp.png")
_images/c603d09466d14dcbacbd76343d5d83d0470db00fdb1a4697bbf5896580b4f450.png 
import yaml
with open(paths.OUT_DIR / "local_gov_ornl_cdiac_coastalms_88w_30n_temp.yaml", "r") as stream:
    stats = yaml.safe_load(stream)
stats

{'bias': {'long_name': 'Bias or MSD',
  'name': 'Bias',
  'value': 32.38266357485453},
 'corr': {'long_name': 'Pearson product-moment correlation coefficient',
  'name': 'Correlation Coefficient',
  'value': nan},
 'descriptive': {'long_name': 'Max, Min, Mean, Standard Deviation',
  'name': 'Descriptive Statistics',
  'value': [22.396244923273724,
   22.371082226435345,
   22.383663574854534,
   0.012581348419189453]},
 'dist': {'long_name': 'Distance in km from data location to selected model location',
  'name': 'Distance',
  'value': 0.12069941685270609},
 'ioa': {'long_name': 'Index of Agreement (Willmott 1981)',
  'name': 'Index of Agreement',
  'value': 0.0},
 'mse': {'long_name': 'Mean Squared Error (MSE)',
  'name': 'Mean Squared Error',
  'value': 1048.6370584925387},
 'rmse': {'long_name': 'Root Mean Square Error (RMSE)',
  'name': 'RMSE',
  'value': 32.38266601891417},
 'ss': {'long_name': 'Skill Score (Bogden 1996)',
  'name': 'Skill Score',
  'value': -inf}}