Starting to Publish Data

Individual researchers and research organizations can use CUAHSI HIS to publish their data as a CUAHSI Water Data Service.  There are four steps to the initial establishment of a CUAHSI Water Data Service:

• Setting up or arranging access to a HIS Server (Server Set-up)
• Storing observations data in the CUAHSI Observations Data Model (ODM) (Data Loading)
• Providing access to this data through WaterOneFlow web services (Network Set-up)
• Indexing the resulting water data service at HIS Central in the national water metadata catalog (Data Indexing)

Any organization or individual is free to using CUAHSI Hydrologic Information System tools to accomplish the first three of these steps, and thus to establish their own water data services.  The fourth step connects the water data service to the HIS Central metadata catalog so that it appears in the data discovery tools and responds to the same data retrieval calls as all of the other data sources in CUAHSI-HIS.

1. Server Set-up

To publish and share your data as part of CUAHSI-HIS, the following server options are available:

• Build your own CUAHSI-HIS Server
  • What you Need to Set-up a New CUAHSI-HIS Server.
  • Get a URL/IP address for the server.
  • Setting Up A New CUAHSI-HIS Server.
• Have HIS Central build a CUAHSI-HIS Server for you
  • Contact Ilya Zaslavsky about building the CUAHSI-HIS Server.
  • Get a URL/IP address for the server.
• Have HIS Central build and host a virtual CUAHSI-HIS Server for you
  • Contact Ilya Zaslavsky.
• Have HIS Central host your data network on one of their CUAHSI-HIS Servers
  • Contact Ilya Zaslavsky about hosting.
  • While you will not need your own CUAHSI-HIS server, you will need to load your data into an ODM database, and the most straight-forward way of doing that requires a computer with MicroSoft SQL Server. Additional software may be needed, depending on exactly which techniques are used to load the data.
• Have another CUAHSI-HIS Server host your data network — Discussion is currently underway with a number of state agencies/regional organizations/academic institutions/other interest groups about setting up HIS Servers in order to host networks for their constituencies.
  • Contact the hosting CUAHSI-HIS Server Manager
  • While you will not need your own CUAHSI-HIS server, you will likely need to load your data into an ODM database, and the most straight-forward way of doing that requires a computer with MicroSoft SQL Server. Additional software may be needed, depending on exactly which techniques are used to load the data.

2. Data Loading

A data manager loads data into an ODM database, and checks that the data loaded correctly and is valid.

• Information and Downloads on ODM Databases

• Tools that can be used to load data into an ODM database:
  • ODM Data Loader
  • ODM Streaming Data Loader (SDL)
  • SQL Server Management Studio
  • Microsoft Excel/Microsoft Access
  • SQL Server Integration Service (SSIS)
  • Many other tools can import data into an ODM SQL Server database, so this list is not exclusive
• Tools that can be used to check the data
  • ODM Tools
• Once the data is loaded, they have been transformed from whatever state they were in, to a standardized format with standardized descriptors in our ODM. Be aware that different sources may be more or less sensitive to their data being "transformed" and information on this should become part of its provenance.

3. Network Set-up

Steps for adding a new network to the system:

• Database — Add the ODM database to the CUAHSI-HIS Server’s SQL Server (if this wasn’t done prior to loading the data into the database).
• DASH — Add the network to DASH (this will have to be refreshed any time a new site is added or the site's lat/long is modified).
• Web Services — Add a new set of web services for the network, creating a WSDL.
• Test — Test to make sure the network is correctly set-up by running HydroExcel to verify that the sites, variables, and data are web service accessible.

4. Data Indexing

Steps for indexing a new network:

• Cataloging — Register Network with HIS Central - As part of this, the user indicates whether the data he registers are frozen, or shall be periodically re-harvested (and at what interval). This determines how often the harvester visits each registered service (or if at all, for frozen data). A moderator examines the submission and approves the WSDL for publication through HydroSeek and approves the user to be able to use the HydroTagger to map variables to ontology concepts. The registration of a WSDL triggers the operation of the sniffer. The sniffer augments the national metadata catalog with the observations catalog derived from this data source.
• Tagging — Run HydroTagger to associate the variables with ontology concepts. The sniffer examined the new WSDL that was listed during the cataloging process and identified any variables that are in the new ODM/WSDL but not already mapped to an ontology concept, and makes a list of these. This list is presented to the user when they log in to HydroTagger so the user can map them.
• Mapping — The dots in the newly created observation network now appear in the national map on HydroSeek. Check that the data can be found in HydroSeek and that it is accessible and downloadable.