You are here: Foswiki>CoreLibrary Web>NVCLGovHackOverview (18 Feb 2016, VincentFazio)Edit Attach

National Virtual Core Library (NVCL) - explained for Melbourne Science Hackfest

Note: If you were interested in the ERML dataset being provided by AuScope for Melbourne Science Hackfest, you can find more information on that at ERMLGovHackOerview

Overview

This page is for participants in Melbourne Science Hackfest who are interested in creating a mashup of the NVCL data. This page provides a quick introduction to the NVCL data, services and its use. If you are looking for the full overview then you can find it at WebHome.

What is NVCL?

The NVCL is best thought of as a library or catalog of borehole (alternatively drillhole) observations. So what is a borehole? Without turning this into a lesson in geoscience, a borehole is simply a narrow hole drilled deep into the ground to allow geoscientists to see what minerals and rocks are there. Unlike your average home drill which would pulverise the rock into dust, these 'borehole drills' extract the drilled material mostly intact. This extracted rock comes out as a long tube of rock known as a borehole core. The borehole core is what geoscientists are really interested in as they provide a narrow snapshot into what lies underneath the ground.

Borehole cores in trays (courtesy of the Geological Survey of Western Australia)

gswa-borehole-core.png

Now these borehole cores (long segments of rock) are typically broken up and assembled into smaller trays before being warehoused somewhere. Originally a geologist would examine the resulting rock core by hand in order to report on what minerals are available at what depths. These days the borehole core is run through a machine known as a hyperspectral logger (HyLogger) which estimates the quantities of minerals along the length of the borehole.

Detected Mineral Quantity vs Borehole Depth (For illustrative purposes only)

borehole-analytes.png

All of this analytical data along with information about the drilling process itself is now completely digital and available through web services, this is the National Virtual Core Library.

So what is available?

Quite a bit actually, It wasn't mentioned before but most of the state/territories are participating in the NVCL program and have public facing web services. This data is exposed through three different endpoints for each provider:
  1. A Web Feature Service For the borehole and observational metadata (where is the borehole, who drilled it)
  2. A NVCL Data service For requesting high resolution imagery and plots of the HyLogger data
  3. A NVCL Download service For requesting the underlying data be bundled up in a variety of formats (This is a queing service, it does not do realtime requests).
As of February 2016 - these are the known NVCL providers and their service endpoints:

CSIRO
WFS - http://nvclwebservices.vm.csiro.au/geoserverBH/wfs
Data service - http://nvclwebservices.vm.csiro.au/NVCLDataServices/
Download service - http://nvclwebservices.vm.csiro.au/NVCLDownloadServices/

Geoscience Western Australia
WFS - http://geossdi.dmp.wa.gov.au/services/wfs
Data service - http://geossdi.dmp.wa.gov.au/NVCLDataServices/
Download service - http://geossdi.dmp.wa.gov.au/NVCLDownloadServices/

Mineral Resources Tasmania:
WFS - http://www.mrt.tas.gov.au/web-services/wfs
Data service - http://www.mrt.tas.gov.au/NVCLDataServices/
Download Service - http://www.mrt.tas.gov.au/NVCLDownloadServices/

Northern Territory Geological Survey
WFS - http://geology.data.nt.gov.au/geoserver/wfs
Data service - http://geology.data.nt.gov.au/NVCLDataServices/
Download service - http://geology.data.nt.gov.au/NVCLDownloadServices/

Primary Industries and Resources South Australia
WFS - https://sarigdata.pir.sa.gov.au/nvcl/geoserver/wfs
Data service - https://sarigdata.pir.sa.gov.au/nvcl/NVCLDataServices/
Download service - https://sarigdata.pir.sa.gov.au/nvcl/NVCLDownloadServices/

Department of Primary Industries New South Wales
WFS - http://auscope.dpi.nsw.gov.au/geoserver/wfs
Data service - http://auscope.dpi.nsw.gov.au/NVCLDataServices/
Download service - http://auscope.dpi.nsw.gov.au/NVCLDownloadServices/

Department of Economic Development, Jobs, Transport and Resources Victoria
WFS - http://geology.data.vic.gov.au/nvcl/wfs
Data service - http://geology.data.vic.gov.au/NVCLDataServices/
Download service - http://geology.data.vic.gov.au/NVCLDownloadServices/

Department of Natural Resources and Mines Queensland
WFS
- http://geology.information.qld.gov.au/geoserver/wfs
Data service - http://geology.information.qld.gov.au/NVCLDataServices/
Download service - http://geology.information.qld.gov.au/NVCLDownloadServices/

How do I access the data?

Well that depends on what you are after, up until now I've been talking about borehole cores, metadata, observational data etc and you've probably assumed they are all related. Well, you are correct, there is a series of relationships between the various instances of data that will require you to navigate the various services. Hopefully I can illustrate these relationships with the following map:

nvcl-map.png

Requesting Boreholes

As a consumer of NVCL you'll typically want to start by querying for Boreholes from the Web Feature Service (WFS), from there you can always provide 'drill down' (pun intended) functionality for more information about the data for that borehole. To get you started, here is a basic WFS query that will request the first 5 boreholes from a WFS as XML:

http://geology.data.vic.gov.au/nvcl/wfs?service=WFS&version=1.1.0&request=GetFeature&typeName=gsml:Borehole&maxFeatures=5

Let's decipher the WFS parameters:
Parameter Purpose
service This should always be 'WFS'
version The version of WFS you wish to query (leave this as 1.1.0)
request WFS services provide a variety of request types, for simplicities sake let's keep it as 'GetFeature'. Read the wfs spec for more information on the others.
typeName A WFS is a collection of feature types (think data types) which can be queried individually. In our case the borehole's have the type name 'gsml:Borehole'
maxFeatures We can limit the size of the response to X features, bigger requests take longer for the server to generate and for you to parse. I'd suggest keeping the request size under 200.
Now WFS is a well defined standard with lots of nuances and particulars that are too varied to cover here so instead we'll focus on the basics. Essentially a provider of a WFS will serve one or more different feature types which can be best thought of as Objects if you are a software developer. That is, each feature type will have a number of attributes and possible 'child' features forming a complex schema. When you make a request for a feature type to a WFS, the service responds with XML encoded instances of that feature type. Eg - If I have a feature type 'City' and query a WFS for all 'City' feature types, I'll get a number of 'City' features (Objects) with names like 'Perth', 'Melbourne', 'Sydney' etc.

If you plan on representing the boreholes spatially (i.e. on a map) then you will be interested in the borehole spatial attribute which can be extracted using the following XPath
#This is relative to the gsml:Borehole element 
#For extracting the point coordinates 
gsml:collarLocation/gsml:BoreholeCollar/gsml:location/gml:Point/gml:pos 

#For extracting the spatial reference system 
gsml:collarLocation/gsml:BoreholeCollar/gsml:location/gml:Point/@srsName

Now because this page is supposed to be a quick introduction to NVCL (and by extension WFS) I'm going to gloss over the parts that involve asking the WFS to filter your boreholes by some attribute. Instead I'm just going to say that you can also make HTTP POST requests to a WFS which can contain filters, if you were to POST the following XML query to any of the above WFS endpoints (sans URL parameters) you would get a response of all boreholes within the specified bounding box (roughly Western Australia) that have a drilling method of 'diamond core'.

POST this to http://geossdi.dmp.wa.gov.au/services/wfs
<?xml version="1.0" encoding="UTF-8"?>
<wfs:GetFeature service="WFS" version="1.1.0" maxFeatures="20" xmlns:wfs="http://www.opengis.net/wfs" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml" xmlns:gsml="urn:cgi:xmlns:CGI:GeoSciML:2.0" xmlns:sa="http://www.opengis.net/sampling/1.0" xmlns:om="http://www.opengis.net/om/1.0" xmlns:nvcl="http://www.auscope.org/nvcl">
    <wfs:Query typeName="gsml:Borehole" srsName="EPSG:4326">
        <ogc:Filter>
            <ogc:And>
                <ogc:BBOX>
                    <ogc:PropertyName>gsml:collarLocation/gsml:BoreholeCollar/gsml:location</ogc:PropertyName>
                    <gml:Envelope srsName="EPSG:4326">
                        <gml:lowerCorner>110 -13 </gml:lowerCorner>
                        <gml:upperCorner>130 -36 </gml:upperCorner>
                    </gml:Envelope>
                </ogc:BBOX>
                <ogc:PropertyIsEqualTo>
                    <ogc:PropertyName>gsml:indexData/gsml:BoreholeDetails/gsml:drillingMethod</ogc:PropertyName>
                    <ogc:Literal>diamond core</ogc:Literal>
                </ogc:PropertyIsEqualTo>
            </ogc:And>
        </ogc:Filter>
    </wfs:Query>
</wfs:GetFeature>

Requesting Datasets

Once you have a borehole you are interested in the next step is requesting the list of datasets to go with that borehole, to do this you must extract the borehole name from the gsml:Borehole feature type (the WFS response). The borehole name is identified by the gml:name element with the attribute codeSpace="http://www.csiro.au". If you are familiar with XPath then you will be able to extract the correct value with the following XPath expression:
wfs:FeatureCollection/gml:featureMembers/gsml:Borehole/gml:name[@codeSpace='http://www.csiro.au']

If you are unfamiliar with XPath, then the following XML snippet should highlight the part you need to parse:

borehole-xml.png

Once you have the borehole name you'll need to query the data service for the list of datasets using the getDatasetCollection request

http://nvclwebservices.vm.csiro.au/NVCLDataServices/getDatasetCollection.html?holeidentifier=WTB5

This will return a simple XML response that looks something like this (please note that you may receive an empty collection or a collection with multiple datasets). You will be reusing the 'DatasetID' element in future queries so make sure you extract it!
<DatasetCollection>
    <Dataset>
        <DatasetID>6dd70215-fe38-457c-be42-3b165fd98c7</DatasetID>
        <DatasetName>WTB5</DatasetName>
        <OmUrl>http://nvclwebservices.vm.csiro.au/geoserverBH/</OmUrl>
    </Dataset>
</DatasetCollection>

Requesting Logged Elements

So now that you have a borehole and dataset that you wish to query, the next step is identifying 'logged elements'. The data service classifies all high resolution imagery and detected analyte (eg Iron, Gold) levels as logged elements. In order to request any of this information you need the ID of the element to request, to discover these ID's you can request all logged elements for a dataset with the following getLogCollection requests:

For high resolution imagery logged elements:

http://nvclwebservices.vm.csiro.au/NVCLDataServices/getLogCollection.html?datasetid=6dd70215-fe38-457c-be42-3b165fd98c7&mosaicsvc=yes

For analyte logged elements:

http://nvclwebservices.vm.csiro.au/NVCLDataServices/getLogCollection.html?datasetid=6dd70215-fe38-457c-be42-3b165fd98c7&mosaicsvc=no

The above requests will both generate a series of XML Log elements containing the logged element ID (important), a descriptive name of the logged element and an optional sample count. The sample count will identify how many images/samples are associated with a particular logged element.

Using logged elements for imagery

The response from the above getLogCollection request (with mosaicsvc=yes) will contain a number of logged elements relating to high resolution images.You can differentiate between the logged elements based on their logName as the following table will demonstrate:
logName What it represents
Mosaic

Low resolution image representing the entire borehole, great as a summary image.

eg: http://nvclwebservices.vm.csiro.au/NVCLDataServices/Display_Tray_Thumb.html?logid=63d31981-bd0c-44dd-a118-6a3406c5d68&sampleno=0

Imagery

High resolution imagery of the borehole broken down into 64 pixel height images. When all images are stacked vertically, forms a clear picture of the borehole.

The response from the getLogCollection will indicate how many images there are to request in the sample count field.

http://nvclwebservices.vm.csiro.au/NVCLDataServices/Display_Tray_Thumb.html?logid=fae8f90d-2015-4200-908a-b30da787f01&sampleno=123

<anything else> Imagery of an undefined type, the logName should be descriptive
There are two ways you can request the logged images:
  1. Query the images directly. Use the logId from the getLogCollection response, the sampleId is 0 based and references to the image index to request
    http://nvclwebservices.vm.csiro.au/NVCLDataServices/Display_Tray_Thumb.html?logid=fae8f90d-2015-4200-908a-b30da787f01&sampleno=123
    http://nvclwebservices.vm.csiro.au/NVCLDataServices/Display_Tray_Thumb.html?logid=fae8f90d-2015-4200-908a-b30da787f01&sampleno=124
    http://nvclwebservices.vm.csiro.au/NVCLDataServices/Display_Tray_Thumb.html?logid=fae8f90d-2015-4200-908a-b30da787f01&sampleno=125
  2. Query the service to generate a HTML page which can be rendered to show a range of images:
    http://nvclwebservices.vm.csiro.au/NVCLDataServices/mosaic.html?logid=fae8f90d-2015-4200-908a-b30da787f01&startsampleno0&endsampleno=100&width=100

Using logged elements for analytes

The logged elements for analytes can be plotted into a graph (raster image) of the data using a plotScalar request:

http://nvclwebservices.vm.csiro.au/NVCLDataServices/plotscalar.html?logid=1da32ce8-32dc-4deb-840f-f619a929ede&startdepth=0&enddepth=99999&samplinginterval=1&graphtype=1&width=300&height=600

This request will plot the variance of the detected analyte as a function of the length of the borehole.
Parameter Effect
logid The logId as extracted from the getLogCollection response
startdepth Where to begin the plot of the graph (in metres) as relative to the top of the borehole.
enddepth Where to end the plot of the graph (in metres) as relative to the top of the borehole
samplinginterval The interval of the sampling (in metres)
width The width of the output image in pixels
height The height of the output image in pixels
graphtype

What type of graph should be returned. It should be an integer in the range of 1 to 3,

1 = Stacked Bar Chart
2 = Scattered Chart
3 = Line Chart

How is it currently being used?

AuScope Portal

The AuScope Portal is a web based GIS portal for displaying spatial data sets like NVCL. It supports the full spectrum of NVCL services, to see it in action please visit the portal and click on any of the blue markers.

AuScope Portal with NVCL layer

auscope-portal-nvcl.png

If you were at all interested in how the portal works then feel free to download the source and play around, info on getting started is available on the AuScope wiki, alternatively have a chat to one of the AuScope data mentors.

- JoshVote - 24 May 2012 -- Updates from JesseRobertson - 07 Jul 2014, -- VincentFazio - 18 Feb 2016
Topic attachments
I Attachment Action Size Date Who Comment
auscope-portal-nvcl.pngpng auscope-portal-nvcl.png manage 528 K 25 May 2012 - 05:41 JoshVote  
borehole-analytes.pngpng borehole-analytes.png manage 244 K 24 May 2012 - 01:12 JoshVote  
borehole-scalar.jpgjpg borehole-scalar.jpg manage 70 K 24 May 2012 - 01:12 JoshVote  
borehole-xml.pngpng borehole-xml.png manage 19 K 24 May 2012 - 05:59 JoshVote  
gswa-borehole-core.pngpng gswa-borehole-core.png manage 322 K 24 May 2012 - 00:51 JoshVote  
nvcl-map.graphmlgraphml nvcl-map.graphml manage 11 K 24 May 2012 - 03:44 JoshVote  
nvcl-map.pngpng nvcl-map.png manage 6 K 24 May 2012 - 03:44 JoshVote  
Topic revision: r12 - 18 Feb 2016, VincentFazio
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback