Posts in Dataversity Public Discussion: Dataversity

The Spatial Ecology wiki is a platform to help people learning open source
tools for spatial analysis, modelling and data processing.
http://www.spatial-ecology.net/dokuwiki/doku.php

Might be of interest?
Cheers
Brent Wood

Hi all,

I produced this version of an assessement tool using the Biodata Management
Guide maturity criteria. I'm sharing it in the hopes others will find it
useful. It serves the same purpose as Dan's System Assessement Form, but we
felt this format would work better for us in terms of ease of use (radio
buttons), graphically representing the status of data management for a given
project, and being able to save the assessment within the project folder.

We are about to run some assessments of a sample set of projects from the
Auckland Council Environmental Services and Research & Evaluation Units using
this tool.

If anyone else chooses to have a go with this version of the assessement tool,
I'd welcome any feedback or suggested improvements.

Cheers,
Briony

Biodata Management Guide - Project Assessment Tool 2015-01-27.xlsx 276.33kb

Hi All,

Appended is the official update from MFE, and the Regional and Unitary
Councils, on Environmental Monitoring and Reporting (EMaR). You may note a
couple of highlights pertaining to our industry:

- Near-real time automated exposure of water quality data onto the LAWA
website;
- Plans for exposure to include facility for potential data consumers to
harvest the data via LAWA; and
- Plans for biodiversity indicator reporting onto LAWA later this year.

The biodiversity indicator(s) to be reported for EMaR, via LAWA is still in the
early stages of planning. There is little I can add accept to say the Biodata
Services Stack team is working with the EMaR team to capitilise on effort.

regards,

Rest of post

EMaRUpdate_Jan15.pdf 705.36kb

Hi James
I am Jochen and leader for the BSS project.
You find more info on the project under:
https://teamwork.niwa.co.nz/display/NZBSS

As part of BSS we aim to develop a prototype national data infrastructure to
facilitate data sharing between multiple providers and users. As part of that
we are basing as much as possible on open standards.
We are currently using DwC/IPT for that put are looking also / in addition to
develop a OGC O&M based approach, as DwC can only handle very simple data
types. Generally, in NZ our approach is to follow as much as possible
international standards for open data exchange and we see OGC and GBIF as the
global leaders.
I do not understand your comment on "not so much targeted at Open Data and
providing an API for open innovation", in my mind building a national data
infrastructure and open standards are the first step for getting data together
and foster innovation. But our project is not addressing data analyses APIs or
so (a'la WPS).

Regards, Jochen

Dr Jochen Schmidt
Chief Scientist - Environmental Information

+64-3-343-8058 | +64-27-298-5886 | 10 Kyle Street, Riccarton, Christchurch |
www.niwa.co.nz<http://www.niwa.co.nz>
[NIWA]<http://www.niwa.co.nz>

Rest of post

imagecf9a31.JPG 20.33kb

there was a whole DOC run exercise on veg classification a couple of years ago
(engaging 'experts' from across the country), (and i was involved in an earlier
one last century!!), but these are not the only schema; a multivariate analysis
of vegetation types in nz has been initiatied by Susan Wiser; and of course
there is the spatially explicit LENZ.
colin

Rest of post

________________________________________
From: dataversity_public_discuss@dataversity.org.nz
[dataversity_public_discuss@dataversity.org.nz] on behalf of james.bibby
[<email obscured>]
Sent: Friday, 16 January 2015 5:02 p.m.
To: dataversity_public_discuss@dataversity.org.nz
Subject: Re: [Dataversity] W3C and OGC to Collaborate to Integrate Spatial Data
on the Web

Hi Brent,

No links yet, but happy to share once they are posted. The documentation
supporting the webservice is still being finalised. Will be publishing them on
our website when the service goes live in March this year.

The Biodiversity Services Stack looks very interesting, I like the approach. I
would be keen to keep across it, particularly the standards applied to the
source data sets. From what I read it looks like its facilitating exchange of
data between respositories, not so much targeted at Open Data and providing an
API for open innovation. Is that a correct interpretation?

I assume that the O&M standard is being applied to species sightings, data that
would make its way up to GBIF. Did you guys also look at Darwin Core as the
standard to apply to this data? Would be interested to know what tipped you in
favour of O&M, might influence our own future decisions.

We are targeting to have two other services go live this year that we have yet
to confirm standards for. Systematic Survey data sets, and a Vegetation
Classfication data set.

EML (http://knb.ecoinformatics.org/software/eml/) looks like a good candidate
for survey metadata, but I have yet to find a vocab or standard approach for
the survey data itself. I have had no luck with Vegetation Classificaiton -
descriptions of plant community types. Any suggestions or thoughts welcome.

James
--

View topic http://dataversity.org.nz/r/topic/1ofsRN5CZzIKh3B0Lm30Bn

Leave group
mailto:dataversity_public_discuss@dataversity.org.nz?Subject=unsubscribe

Start groups http://OnlineGroups.net

________________________________

Please consider the environment before printing this email
Warning: This electronic message together with any attachments is confidential.
If you receive it in error: (i) you must not read, use, disclose, copy or
retain it; (ii) please contact the sender immediately by reply email and then
delete the emails.
The views expressed in this email may not be those of Landcare Research New
Zealand Limited. http://www.landcareresearch.co.nz

Hi Brent,

No links yet, but happy to share once they are posted. The documentation
supporting the webservice is still being finalised. Will be publishing them on
our website when the service goes live in March this year.

The Biodiversity Services Stack looks very interesting, I like the approach. I
would be keen to keep across it, particularly the standards applied to the
source data sets. From what I read it looks like its facilitating exchange of
data between respositories, not so much targeted at Open Data and providing an
API for open innovation. Is that a correct interpretation?

I assume that the O&M standard is being applied to species sightings, data that
would make its way up to GBIF. Did you guys also look at Darwin Core as the
standard to apply to this data? Would be interested to know what tipped you in
favour of O&M, might influence our own future decisions.

We are targeting to have two other services go live this year that we have yet
to confirm standards for. Systematic Survey data sets, and a Vegetation
Classfication data set.

EML (http://knb.ecoinformatics.org/software/eml/) looks like a good candidate
for survey metadata, but I have yet to find a vocab or standard approach for
the survey data itself. I have had no luck with Vegetation Classificaiton -
descriptions of plant community types. Any suggestions or thoughts welcome.

James

Hi James,
We are working on a "Biodata Services Stack", initial pages about this are
here:New Zealand Biodata Services Stack (BSS) - New Zealand Biodata Services
Stack - Teamwork In general, we are looking at OGC Web Services as our
preferred approach, using O&M (Observations & Measurements) for any
spatial/environmental data (along with WFS & with SOS for time series data)
also Darwin Core, with tools like GBIF's IPT for biodata dataset harvesting.
http://www.opengeospatial.org/standards/omhttp://ipt.gbif.org/
I'm not aware of much formal take-up of OASIS standards in science data in
NZ... which may tell more about my ignorance than the real state of affairs...
but SAMS, CIQ, etc are covered in NZ Govt ICT guidelines.
Do you have any links describing your OASIS based approach? Given OASIS is a
member of OGC, I would hope there is some consistency & interoperability there.
I know there is an active group looking at OASIS security models for OGC Web
Services (http://www.opengeospatial.org/projects/groups/geoxacmlswg)

Cheers,
Brent Wood

From: james.bibby <<email obscured>>
To: dataversity_public_discuss@dataversity.org.nz
Sent: Monday, January 12, 2015 4:09 PM
Subject: Re: [Dataversity] W3C and OGC to Collaborate to Integrate Spatial
Data on the Web

Along the international theme (for an expat kiwi), I have been managing a
project to expose biodiversity data held by the Office of Environment &
Heritage here in NSW. I can very closely relate to the benefit of
international collaboration around standards, both for the underlying data and
protocols to exchange that data. We ended up selecting the OData protocol for
data access (https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=odata)
which will be common across all our services, and for the first service to go
live (species sightings) we used Darwin Core.

One of our projects key considerations,which is referenced in that OGC
statement, is how do we reduce the cost and complexity for people wishing
integrating our data with their systems or new apps. In particular, making
development effort reusable across other data providers by adopting a
non-proprirary API for our data.

This was the basis for trying to go down the OData path. I would be interested
in understanding if this is an issue also being tackled in NZ, and if it is
being addressed through standards?

James

Rest of post

Along the international theme (for an expat kiwi), I have been managing a
project to expose biodiversity data held by the Office of Environment &
Heritage here in NSW. I can very closely relate to the benefit of
international collaboration around standards, both for the underlying data and
protocols to exchange that data. We ended up selecting the OData protocol for
data access (https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=odata)
which will be common across all our services, and for the first service to go
live (species sightings) we used Darwin Core.

One of our projects key considerations,which is referenced in that OGC
statement, is how do we reduce the cost and complexity for people wishing
integrating our data with their systems or new apps. In particular, making
development effort reusable across other data providers by adopting a
non-proprirary API for our data.

This was the basis for trying to go down the OData path. I would be interested
in understanding if this is an issue also being tackled in NZ, and if it is
being addressed through standards?

James

May be of interest, despite the time zone...
Brent Wood
----- Forwarded Message -----
From: Kathy Martinolich - NOAA Affiliate <<email obscured>>
To: "<email obscured>" <<email obscured>>
Sent: Friday, January 9, 2015 10:28 AM
Subject: [metadata] Metadata for Managers Course Beginning January 13th

Metadata for Managers

This course presents the concept, principles, and value of metadata. Course
modules teach core metadata concepts and present the business case
including policy and implementation guidance, for using metadata. The
course provides targeted instruction to managers who wish to understand the
who? what? where? and why? of metadata. Metadata for Managers is not
limited strictly to 'managers,’ but may be taken as a basic familiarity
course or refresher course for anyone wishing to take additional courses
dealing with more advanced metadata topics.

The course will be presented in an online, webinar format on Tuesdays, at
9 AM Central Standard Time.

Course modules offered include:

January 13 - Metadata 101
January 20 - Metadata Policies
January 27 - Content
February 3 - Discovery
February 9 - Implementation
February 16 - Question and Answers

To Register, please use this link:
https://attendee.gotowebinar.com/register/120602262

For other courses offered, please see
http://www.ncddc.noaa.gov/metadata-standards/metadata-training/

Contact <email obscured> if you have questions.

Hope this helps!
Kathy

Rest of post

An interesting start to the year....

Data sharing & reuse initiatives in NZ might benefit from some international
work going on regarding standards & protocols and tools (many open source) to
improve discovery, delivery & interoperability.
This follows an international meeting discussing these issues last March where
issues included linked data & managing crowd sourced data alongside "official"
data, regarding quality issues etc. (I don't think NZ was represented? Linking
Geospatial Data, 5th - 6th March 2014, London (Report) )

The official OGC statement is at
http://www.opengeospatial.org/pressroom/pressreleases/2149

The open data domain seems to be following a consistent paradigm:
"useful open data systems require effective open standards, which are often a
better fit with open source than proprietary solutions."

Cheers
Brent Wood

Thanks Brent, that looks most useful. I hope one of us will attend... Onto it.

Elise

Rest of post

-----Original Message-----
From: Brent Wood
Sent: 16/12/2014 09:32
To: Alison - Work; NZOSS Open Discussion List;
dataversity_public_discuss@dataversity.org.nz; <email obscured>;
<email obscured>; Jeremy Palmer; Elise Smith;
<email obscured>
Subject: [Dataversity] Postgres replication tutorial

If you are interested in robust data storage using the Postgres database
management system - read on.

Core NZ Govt agencies are required to make high value public data freely
available. Other agencies such as CRI's, councils and universities are
encouraged to do the same. The first step in providing access to data is to
implement robust and secure systems to manage these data. NIWA, like many other
organisations, is using the free, open source Postgresql RDBMS to do this,
generally in conjunction with Postgis for spatial data support.

A robust data service requires some level of redundancy, if a database server
fails, ideally a remote standby system is waiting to take over so the service
continues with customers unaware of any problems.

The topic is also relevant for any organisation looking to ensure their data or
data services are well managed, including businesses, NGO's, councils, etc.

PG Experts consultant and Postgres developer Josh Berkus will be in New Zealand
for LinuxConfAU in January. NIWA is taking advantage of this opportunity to
have Josh run a workshop in Wellington on Postgres database replication, which
will have places for non-NIWA staff.

The workshop will be on Jan 20, 2015, at NIWA, Wellington (Greta Point)
starting at 9:00AM, and finishing by 3:00PM. There is a charge of $200 per
person.
Email <email obscured> to question or book a place - I'm currently at sea,
with limited internet/email access, but will eventually get any messages.

In Josh's words:

PostgreSQL’s built-in replication is conceptually simple, but the devil
is in the details, and this tutorial is all about the details. We will
cover the core essentials of how to set up and configure binary
replication, using hands-on exercises for some elements, including:

* setup & configuration
* replication security
* cascading replication
* synchronous replication
* failover and failback
* replication management tools
* performance tuning replication

Materials and Docker test environments will be updated at the
pgReplicationTutorial Github project:
(https://github.com/jberkus/pgReplicationTutorial) two weeks before the
event. Attendees are strongly encouraged to download it and set up the
learning environment before the tutorial; time will not be given during
the tutorial to do so.

Attendees will leave knowing how to set up a simple replication cluster,
and about the options for more complex configurations.

--

View topic http://dataversity.org.nz/r/topic/75emfvNz8P4gztBF9M4gvz

Leave group
mailto:dataversity_public_discuss@dataversity.org.nz?Subject=unsubscribe

Start groups http://OnlineGroups.net

If you are interested in robust data storage using the Postgres database
management system - read on.

Core NZ Govt agencies are required to make high value public data freely
available. Other agencies such as CRI's, councils and universities are
encouraged to do the same. The first step in providing access to data is to
implement robust and secure systems to manage these data. NIWA, like many other
organisations, is using the free, open source Postgresql RDBMS to do this,
generally in conjunction with Postgis for spatial data support.

A robust data service requires some level of redundancy, if a database server
fails, ideally a remote standby system is waiting to take over so the service
continues with customers unaware of any problems.

The topic is also relevant for any organisation looking to ensure their data or
data services are well managed, including businesses, NGO's, councils, etc.

PG Experts consultant and Postgres developer Josh Berkus will be in New Zealand
for LinuxConfAU in January. NIWA is taking advantage of this opportunity to
have Josh run a workshop in Wellington on Postgres database replication, which
will have places for non-NIWA staff.

The workshop will be on Jan 20, 2015, at NIWA, Wellington (Greta Point)
starting at 9:00AM, and finishing by 3:00PM. There is a charge of $200 per
person.
Email <email obscured> to question or book a place - I'm currently at sea,
with limited internet/email access, but will eventually get any messages.

In Josh's words:

PostgreSQL’s built-in replication is conceptually simple, but the devil
is in the details, and this tutorial is all about the details. We will
cover the core essentials of how to set up and configure binary
replication, using hands-on exercises for some elements, including:

* setup & configuration
* replication security
* cascading replication
* synchronous replication
* failover and failback
* replication management tools
* performance tuning replication

Materials and Docker test environments will be updated at the
pgReplicationTutorial Github project:
(https://github.com/jberkus/pgReplicationTutorial) two weeks before the
event. Attendees are strongly encouraged to download it and set up the
learning environment before the tutorial; time will not be given during
the tutorial to do so.

Attendees will leave knowing how to set up a simple replication cluster,
and about the options for more complex configurations.

Rest of post

Hi Brent,
just came across this conversation again whilst looking for some
other info. Reminded me a of a few bits and pieces I;ve come across

https://www.seegrid.csiro.au/wiki/Infosrvices/CKANHarvestingGuide

https://github.com/geosolutions-it/ckanext-geonetwork

http://osgeo-org.1560.x6.nabble.com/CKAN-csw-Metadata-Harvesting-td5073100.html

For completeness, blog post I've previously mentioned
http://geo-solutions.blogspot.it/2013/12/OpenDataNetwork.html and your follow
up note
http://groups.open.org.nz/groups/ninja-talk/messages/topic/2p4w7yDqRbNqXLvRctSjpx

I've not really looked into this - so not advocating for it, just making people
aware of work integrating CKAN and Geonetwork.

Cheers
Aaron
PS We have a few records (actually more than are showing, but a number are
currently marked private) in our public CKAN install
http://datastore.landcareresearch.co.nz/

Thanks Dan,

Thats funny! I saw that xkcd cartoon when it came out. Its impressive that
Flickr rose to the challenge.

The high tech near-future is going to be able to do all sorts of amazing things
with the date-time stamped and geotagged photos we take today. Imagine what we
could have done with millions of such images had they been taken 100 years ago.

I can see the NatureWatch NZ of the future identifying uploaded photos and
sounds all by itself, with crowdsourcing only needed for validation.

Keep on sharing those geotagged images.

Cheers,

Jon
https://www.flickr.com/photos/mollivan_jon/

Rest of post

________________________________________

Jon Sullivan, Ph.D.

Senior Lecturer

Department of Ecology

P.O. Box 84

Lincoln University / Te Whare Wanaka O Aoraki

Lincoln 7647

New Zealand

office: Room 436a, Burns building, Lincoln University campus

Cnr Springs & Ellesmere Junction Roads

email: <email obscured><mailto:Jon.Sullivan@lincoln.ac.nz>

tel: (03) 423 0756 *note the new number*

for international calls, replace (03) with (643)

profile: http://www.lincoln.ac.nz/staff-profile?staffId=Jon.Sullivan

my blog: http://jonsullivan.canterburynature.org
my observations on NatureWatch NZ: http://naturewatch.org.nz/observations/jon
Ecology Department blog: http://EcoLincNZ.blogspot.com

campus Fulbright advisor: ask me about opportunities for US study and research

____________________________________________________

On 21/10/2014, at 10:12 am, Dan Randow
<<email obscured><mailto:dan@randow.co.nz>> wrote:

Flickr has figured out how to tell if an image is of a bird or not,
using a deep convolutional neural network
<http://code.flickr.net/2014/10/20/introducing-flickr-park-or-bird/>

Seems to work pretty well, and did not take a research team and five years.

Dan

--
Dan Randow
Collaboration Wrangler
http://randow.co.nz
+64 27 431 4928

--

View topic http://dataversity.org.nz/r/topic/2pS9ZTCxjVcbndUolJvneZ

Leave group
mailto:dataversity_public_discuss@dataversity.org.nz?Subject=unsubscribe

Start groups http://OnlineGroups.net

________________________________
P Please consider the environment before you print this email.
"The contents of this e-mail (including any attachments) may be confidential
and/or subject to copyright. Any unauthorised use, distribution, or copying of
the contents is expressly prohibited. If you have received this e-mail in
error, please advise the sender by return e-mail or telephone and then delete
this e-mail together with all attachments from your system."

That'd be swell. Here are a few refs to get you started:

Aerts S, Haeussler M, van Vooren S, Griffith OL, Hulpiau P, Jones SJ,
Montgomery SB, Bergman CM, Open Regulatory Annotation Consortium. Text-mining
assisted regulatory annotation
<http://genomebiology.com/content/9/2/R31> *Genome
Biol*. 2008;9(2):R31. PMID: 18271954
<http://www.ncbi.nlm.nih.gov/pubmed/18271954> PMC: PMC2374703
<http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2374703/>

Haeussler M, Gerner M, Bergman CM. Annotating genes and genomes with DNA
sequences extracted from biomedical articles
<http://bioinformatics.oxfordjournals.org/content/27/7/980.long>
*Bioinformatics*. 2011 Apr 1;27(7):980-6. PMID: 21325301
<http://www.ncbi.nlm.nih.gov/pubmed/21325301> PMC: PMC3065681
<http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3065681/>

Van Noorden R. Trouble at the text mine
<http://www.nature.com/news/trouble-at-the-text-mine-1.10184> *Nature*.
2012 Mar 7;483(7388):134-5.

Rest of post

----------------------------
Paul P. Gardner
Rutherford Discovery Fellow
University of Canterbury
www.ucbioinformatics.org
----------------------------

On Tue, Oct 21, 2014 at 11:47 AM, Edward Abraham <<email obscured>>
wrote:

> We could try!
>
> On 21 October 2014 11:31, Paul P. Gardner <<email obscured>>
> wrote:
> > Neat. If we could get accurate OCR for bitmapped images of sequence
> > alignments I'd be a happy camper.
> >
> > ----------------------------
> > Paul P. Gardner
> > Rutherford Discovery Fellow
> > University of Canterbury
> > www.ucbioinformatics.org
> > ----------------------------
> >
> > On Tue, Oct 21, 2014 at 10:12 AM, Dan Randow <<email obscured>> wrote:
> >
> >>
> >> Flickr has figured out how to tell if an image is of a bird or not,
> >> using a deep convolutional neural network
> >> <http://code.flickr.net/2014/10/20/introducing-flickr-park-or-bird/>
> >>
> >> Seems to work pretty well, and did not take a research team and five
> years.
> >>
> >> Dan
> >>
> >> --
> >> Dan Randow
> >> Collaboration Wrangler
> >> http://randow.co.nz
> >> +64 27 431 4928
> >>
> >>
> >> --
> >>
> >> View topic http://dataversity.org.nz/r/topic/2pS9ZTCxjVcbndUolJvneZ
> >>
> >> Leave group mailto:dataversity_public_discuss@dataversity.org.nz
> >> ?Subject=unsubscribe
> >>
> >> Start groups http://OnlineGroups.net
> >>
> >> This email may be confidential and subject to legal privilege, it may
> >> not reflect the views of the University of Canterbury, and it is not
> >> guaranteed to be virus free. If you are not an intended recipient,
> >> please notify the sender immediately and erase all copies of the message
> >> and any attachments.
> >>
> >> Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
> >> information.
> >>
> >
> > --
> >
> > View topic http://dataversity.org.nz/r/topic/4r3iCkJTVGPUvWnd1EnhUz
> >
> > Leave group mailto:dataversity_public_discuss@dataversity.org.nz
> ?Subject=unsubscribe
> >
> > Start groups http://OnlineGroups.net
>
>
>
> --
>
> Edward Abraham
> www.dragonfly.co.nz
> Dragonfly Science, PO Box 27535, Wellington 6141, New Zealand
> Level 5, 158 Victoria Street, Te Aro, Wellington
> M: +64 21 989 454
> T: +64 4 385 9285
>
> --
>
> View topic http://dataversity.org.nz/r/topic/7f8rPvZhLrlOLjP1sIoYxE
>
> Leave group mailto:dataversity_public_discuss@dataversity.org.nz
> ?Subject=unsubscribe
>
> Start groups http://OnlineGroups.net
>
> This email may be confidential and subject to legal privilege, it may
> not reflect the views of the University of Canterbury, and it is not
> guaranteed to be virus free. If you are not an intended recipient,
> please notify the sender immediately and erase all copies of the message
> and any attachments.
>
> Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
> information.
>

We could try!

On 21 October 2014 11:31, Paul P. Gardner <<email obscured>>
wrote:
> Neat. If we could get accurate OCR for bitmapped images of sequence
> alignments I'd be a happy camper.
>
> ----------------------------
> Paul P. Gardner
> Rutherford Discovery Fellow
> University of Canterbury
> www.ucbioinformatics.org
> ----------------------------
>
> On Tue, Oct 21, 2014 at 10:12 AM, Dan Randow <<email obscured>> wrote:
>
>>
>> Flickr has figured out how to tell if an image is of a bird or not,
>> using a deep convolutional neural network
>> <http://code.flickr.net/2014/10/20/introducing-flickr-park-or-bird/>
>>
>> Seems to work pretty well, and did not take a research team and five years.
>>
>> Dan
>>
>> --
>> Dan Randow
>> Collaboration Wrangler
>> http://randow.co.nz
>> +64 27 431 4928
>>
>>
>> --
>>
>> View topic http://dataversity.org.nz/r/topic/2pS9ZTCxjVcbndUolJvneZ
>>
>> Leave group mailto:dataversity_public_discuss@dataversity.org.nz
>> ?Subject=unsubscribe
>>
>> Start groups http://OnlineGroups.net
>>
>> This email may be confidential and subject to legal privilege, it may
>> not reflect the views of the University of Canterbury, and it is not
>> guaranteed to be virus free. If you are not an intended recipient,
>> please notify the sender immediately and erase all copies of the message
>> and any attachments.
>>
>> Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
>> information.
>>
>
> --
>
> View topic http://dataversity.org.nz/r/topic/4r3iCkJTVGPUvWnd1EnhUz
>
> Leave group
mailto:dataversity_public_discuss@dataversity.org.nz?Subject=unsubscribe

Rest of post

Neat. If we could get accurate OCR for bitmapped images of sequence
alignments I'd be a happy camper.

Rest of post

Flickr has figured out how to tell if an image is of a bird or not,
using a deep convolutional neural network
<http://code.flickr.net/2014/10/20/introducing-flickr-park-or-bird/>

Seems to work pretty well, and did not take a research team and five years.

Rest of post

The National Vegetation Survey Databank (NVS) is a physical archive and
electronic databank containing records of over 94,000 vegetation survey plots -
including data from over 19,000 permanent plots. Each year the NVS team
produces an annual report to summarise the activities for the year. This
report describes the new data that has been archived in NVS, significant data
revisions that have been made, efforts to increase end-user awareness, how NVS
data are being used by data accumulators (such as GBIF), international
collaborations, website usage statistics, a summary of NVS data requests and a
list of publications that are based on data archived in NVS. The 2013-14
report can be accessed from the NVS website via the home page (see "What's
new") or directly from
https://nvs.landcareresearch.co.nz/Resources/Publications. The NVS team
welcomes any feedback about this report.

Regards,

Susan Wiser
Scientist, Plant Ecology
Landcare Research
PO Box 69040, Lincoln 7640
NEW ZEALAND

Secretary
International Association for Vegetation Science
www.iavs.org

Rest of post

Hi All,

FWIW, I agree with Andrew. Federation is a great idea. Some collaborators
of mine proposed a system for federating genome annotation many years ago (
http://www.biomedcentral.com/1471-2105/2/7/) Sadly, keeping remote
locations stable, up to date, accurate and consistent is a major hurdle.
The "monolithic data warehouses" still reign supreme for most genome work
(e.g. The UCSC Genome Browser, ENSEMBL). It seems that consistency,
extensive curation and regular updates provided by monoliths continue to
dominate. You can test this by comparing the citation rates for DAS vs e.g.
ENSEMBL.

Best wishes,
Paul.

Rest of post

----------------------------------------------
Paul P. Gardner, Senior Lecturer (Bioinformatics),
Rutherford Discovery Fellow,
School of Biological Sciences <http://www.biol.canterbury.ac.nz/>
University of Canterbury, New Zealand
----------------------------------------------
Phone: +64 (0)3 364 2987 ext. 6742
Email: <email obscured>
----------------------------------------------
Staff page. <http://tinyurl.com/ucppgardner> GoogleScholar.
<http://scholar.google.co.nz/citations?user=yag55pAAAAAJ&;hl=en> *R
<http://rfam.xfam.org/> <http://rfam.xfam.org/>
Assistant Editor in Chief, RNA Biology. <http://tinyurl.com/rnafamily>
Software Editor, PLOS Computational Biology.
<http://www.ploscompbiol.org/static/edboard.action>
----------------------------------------------

On Tue, Sep 2, 2014 at 9:11 AM, Andrew Watkins <<email obscured>>
wrote:

> -- Attachment links are at the end of this email --
>
> my 2d worth
> From my initial scanning dat has some good features. I didnt see it
> particularly as a centralised data approach. rather a toolkit of non
> blocking, streaming data transformation functions. Its very Functional
> Programming and nodejs influenced model and corresponds to some extent in
> my thinking about a purely pipeline driven climate database.
>
> To that extent dat appears to be more pipes than stores.
>
> Out own experience with ingesting data is that practically all primary
> sources of data are horribly flawed, error laden and weakly described. All
> our ingestion programmes are heavy on metadata and type checking,
> validation, error reporting and quality rules. Once you have all that the
> underlying ETL ends up being fairly simple. However our existing ingestion
> processes do tend to take time to run and either use lots of memory or
> have to be batch processed. Similarly when using SOS on large time frames
> we have to wait for all the XML data to arrive before we can start
> displaying it. Browsers can timeout or simply run out of memory so again a
> streaming approach might be useful.
>
> Andrew
>
> On 1/09/14 7:10 pm, "Brent Wood" <<email obscured>> wrote:
>
> >Hi Dan,
> >
> >I've looked at it earlier in the year. You may be interested to note that
> >NIWA's OBIS node was also established with Sloan Foundation funding...
> >they get fingers in lots of pies - but that was also about harvesting
> >from institutional to regional to global repositories (SW Pacific OBIS
> >node to OBIS to GBIF).
> >
> >
> >Apart from dat only being in very early stages of development...
> >
> >
> >Generally in NZ we have focused on a federated approach to data sharing
> >with mostly institutional repositories, made available (& discoverable)
> >via web services. This in contrast to the monolithic data warehouse
> >approach followed by dat.
> >
> >I believe (& your work on data management maturity benchmaking might
> >provide tools to prove or disprove this) that most agencies in NZ are
> >still trying to get their own houses in order, & trying to build a dat
> >system on top of poorly managed & documented datasets is not likely to be
> >that useful.
> >
> >If we can all agree on data delivery & discovery standards, & provide
> >guidelines & tools to help institutions get up to speed to do this with
> >their holdings, then a dat style data warehouse, harvesting from many
> >repositories via common standards with common vocabularies become a very
> >achievable goal.
> >
> >For example - in the hydrometric/climate data domain, various agencies
> >capture rainfall data. This might be a 24hr measurement, but calender day
> >or 0900-0900??? (eg: if manual gauges are read at 0900). In the
> >institutional datasets, the reading may be called "rainfall-24",
> >"rainfall_24hr", "precipitation", etc... So simply harvesting such data
> >into a dat style repository does not necessarily help users know if their
> >mashups are apples & apples, or not...
> >
> >In general, I believe that data needs to be well managed at the
> >institutional level, according to commonly agreed standards, discovered &
> >accessed via agreed, standard protocols with common vocabularies, as a
> >precursor to any monolithic repository. The simplest way to do this is by
> >creating a standardised, federated system, which then allows the various
> >datasets to be usefully harvested into a central system, like a dat
> >repository, if this is regarded as useful.
> >
> >So I think dat is a possible tool for the future, but to start looking at
> >national repositories, when our institutional ones still need work may be
> >trying to run before we can walk - despite the efforts by the dat
> >development team to address these issues.
> >
> >
> >Cheers,
> >
> > Brent Wood
> >
> >
> >
> >
>
> Andrew Watkins
> Systems Development Manager
>
> +64-9-375-2069 | +64-27-703-1007 | 41 Market Place, Viaduct Harbour,
> Auckland | www.niwa.co.nz<http://www.niwa.co.nz>
> [NIWA]<http://www.niwa.co.nz>
> ________________________________
> > From: Dan Randow <<email obscured>>
> >To: dataversity_public_discuss@dataversity.org.nz
> >Sent: Monday, September 1, 2014 3:54 PM
> >Subject: [Dataversity] dat
> >
> >
> >
> >Is anyone looking at the use of dat <http://dat-data.com/> for biodata?
> >
> >dat appears to be a project to build tools to store, synchronize,
> >manipulate and collaborate on sets of data. You can stream data in and
> >out of dat in arbitrary formats.
> >
> >It looks as if it could be useful for biodata federation. One of the
> >team, Bruno Vieira, is a bioinformatician <http://dat-data.com/team.html
> >.
> >
> >Dan
> >
> >--
> >@danrandow
> >Chief Wrangler
> >http://onlinegroups.net
> >64 3 377 5377
> >
> >--
> >
> >View topic http://dataversity.org.nz/r/topic/1vroFgN8hGfLiBUE8YJHRb
> >
> >Leave group
> >mailto:dataversity_public_discuss@dataversity.org.nz?Subject=unsubscribe
> >
> >Start groups http://OnlineGroups.net
> >--
> >
> >View topic http://dataversity.org.nz/r/topic/7LsxZGzcio8TQs6106KbdX
> >
> >Leave group
> >mailto:dataversity_public_discuss@dataversity.org.nz?Subject=unsubscribe
> >
> >Start groups http://OnlineGroups.net
> Dataversity Public Discussion now contains the following file
>
>
> http://dataversity.org.nz/r/file/2YYp1Edo0ZKomoHBnM9Kzvy2usE-5pN-2tRCjgb
> Name: imagea4a9cb.JPG
> Type: image/jpeg
> Size: 20KB
>
> --
>
> View topic http://dataversity.org.nz/r/topic/3z53erdmRC7Aj0VRhVxTJD
>
> Leave group mailto:dataversity_public_discuss@dataversity.org.nz
> ?Subject=unsubscribe
>
> Start groups http://OnlineGroups.net
>
> This email may be confidential and subject to legal privilege, it may
> not reflect the views of the University of Canterbury, and it is not
> guaranteed to be virus free. If you are not an intended recipient,
> please notify the sender immediately and erase all copies of the message
> and any attachments.
>
> Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
> information.
>