News
Data Packaging Tool Released
The TARDIS Data Packaging tool is now available, along with a comprehensive guide on the deposit page. The tool is cross platform (Windows/OS X/Linux) and allows the easy creation of experiment 'packages' for display on TARDIS.edu.au. A user simply points to their data, adds simple annotations and the Packaging Tool takes care of the metadata creation, including metadata automatically extracted from the diffraction images themselves! The resulting package is fit for hosting anywhere (TARDIS' federated storage model), and registration on this site, including persistent handle generation for citation in papers.
Welcome to TARDIS v2
TARDIS Framework v2 is based on simple and free HTTP and FTP services. This eliminates the need for complex and difficult to configure digital repository software, and ensures storage of data for any lab is an easy task to achieve.
Crystallography datasets are no longer archived, meaning one can access individual files without the need to download the whole set. An FTP download feature (optional) allows users to download entire Datasets and Experiments in a single click.
Persistent identifiers provide permanent links to online data, ideal for citation in journal publications. The Australian National Data Service (ANDS) has provided TARDIS with a persistent identifier service, automatically giving users a handle in which to reference their data on experiment registration (coming soon).
The storage of metadata follows the Metadata Encoding and Transmission Standard (METS) profile and the Science and Technology Facilities Council (STFC) data model, for easy communication and translation between different data management systems. As a result, TARDIS is well placed for integration with future systems.
A metadata extraction/data organisation application is currently being developed - making the process of registering data with TARDIS v2 simple. For those that would like to register their data now, don't hesitate to contact steve.androulakis@med.monash.edu.au for assistance.
Crystallography datasets are no longer archived, meaning one can access individual files without the need to download the whole set. An FTP download feature (optional) allows users to download entire Datasets and Experiments in a single click.
Persistent identifiers provide permanent links to online data, ideal for citation in journal publications. The Australian National Data Service (ANDS) has provided TARDIS with a persistent identifier service, automatically giving users a handle in which to reference their data on experiment registration (coming soon).
The storage of metadata follows the Metadata Encoding and Transmission Standard (METS) profile and the Science and Technology Facilities Council (STFC) data model, for easy communication and translation between different data management systems. As a result, TARDIS is well placed for integration with future systems.
A metadata extraction/data organisation application is currently being developed - making the process of registering data with TARDIS v2 simple. For those that would like to register their data now, don't hesitate to contact steve.androulakis@med.monash.edu.au for assistance.
TARDIS 2 Beta Released
The second version of TARDIS has been released into beta:
TARDIS 2 provides the ability to register crystallography data based on simple federated web server storage. We hope that breaking ties with custom digital repository storage in favour of simpler and easier to set up technologies will spur labs into depositing their data with us.
More information in the TARDIS v1 wiki: http://tardis.edu.au/wiki/index.php/TARDIS_Web_Stores
While an automated tool for organising data for registration in TARDIS v2 is almost finished development, users who wish to store data right now can contact steve.androulakis@med.monash.edu.au and help will be happily provided.
Google Group created
A Google Group has been created to facilitate discussion on TARDIS, and where it's headed in the future.
TARDIS | Google Groups
TARDIS | Google Groups
Wiki Created
A wiki has been created to hold information on future development, foster collaboration and feedback from the crystallography community, metadata experts, repository developers/administrators and anyone else who is interested in our progress.
Wiki
Wiki
DatasetTools Update
A minor update has been made to DatasetTools, with a fix to Oscillation Range extraction, and a slight change of schema.
Please update your version by re-downloading the software on the Tools page.
Please update your version by re-downloading the software on the Tools page.
TARDIS Portal Out Of Beta
Currently harvesting datasets out of the Monash Digital Repository is the TARDIS Portal, which was just brought out of beta.
To see the current indexed datasets, see: Data Portal.
To see the current indexed datasets, see: Data Portal.
TARDIS Beta Portal
Work on the very first version of the TARDIS portal is in beta. It currently scans the Monash Test Repository for crystallography datasets and harvests their information for searching/downloading from this site.
Data Portal Beta
Data Portal Beta
Acta Cryst. Paper Online
Our Acta Cryst paper, "Federated repositories of X-ray diffraction images" is now online!
http://www.ncbi.nlm.nih.gov/pubmed/18566516?ordinalpos=1&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
http://www.ncbi.nlm.nih.gov/pubmed/18566516?ordinalpos=1&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Dataset Tools Released
The Dataset Tools suite has been released.
Download
Additionally, a tutorial video has been created demonstrating the tools:
Tutorial Video
Download
Additionally, a tutorial video has been created demonstrating the tools:
Tutorial Video
TARDIS Mentioned In Acta Cryst
The TARDIS web site has been mentioned in the Acta Crystallographica Section D editorial this month.
Find it here: http://scripts.iucr.org/cgi-bin/paper?S0907444908004915
Find it here: http://scripts.iucr.org/cgi-bin/paper?S0907444908004915
Site Revamp
The site's gone through some drastic cosmetic changes in the last couple of days in preparation for the release of Dataset Tools. The tools are considered 'done' and will be released very soon. For now, there are screenshots and descriptions on the Tools page.
There's also an updated metadata schema for datasets, coupled with an example produced using Dataset Tools.
There's also an updated metadata schema for datasets, coupled with an example produced using Dataset Tools.
Deposition Tools Coming Soon!
Update: Dataset Tools has been released
A set of desktop tools that will make depositing crystallography datasets into a Fedora 2.2.1 repository easier are about to be released in beta on this site. Fedora is the fundamental persistent storage technology behind digital repository software such as Arrow and Fez.
Each tool was programmed in Java for cross-platform compatibility, each having a GUI version as well as command-line functionality. Together they aim to make it easier for researchers to package their data, create valid metadata, both technical and repository-based, upload the data and then unpackage it again once downloaded. The GUI tools are:
RepositoryPackager
Created to take a set of diffraction images, tar archive them, bzip2 compress them and then split the resultant file on a chosen file size. This is done in an effort to make data more space efficient in a repository, and because large files (over 2GB) were found to crash various server software in the upload/storage process.
Uses the apache tar/bzip2 java libraries and incorporates the accompanying command-line tools TarBzipper and FileSplitter.
METSManager
Fedora repositories ingest METS XML files to create and describe entries being stored. This program uses the Harvard METS Java Toolkit to create a fedora-compatible METS package that includes entered values for ingestion. Currently, DublinCore data is created within the METS package to describe basic values such as the title of the object in the repository and the authors. Technical xml metadata relating to the experiment itself is currently put as plain text in the Description field.
Created METS XML files can be validated against the Fedora-compatible METS schema to test their eligibility for ingestion.
Future plans include the embedding of technical XML data such as that described in the data section of this site for data harvesting.
Is a GUI implementation of the accompanying command-line tool METSCreator.
DataDepositor
Once data is organised for upload, and a METS XML file is created, the object and its data are ready to be ingested and uploaded into the repository. This program makes such a process simple by scanning a supplied directory (non-recursive since the structure of repository objects are flat) and uploading each file into a new object described by the METS XML file.
Uses the fedora-management API, and is a GUI implementation of the accompanying command-line tool of the same name.
Currently only compatible with Fedora 2.2.1 repositories.
RepositoryUnPackager
Once a split set of files created by RepositoryPackager are downloaded again from the repository, they can be re-joined, uncompressed and unarchived to restore them to their original form using this program.
Is a GUI implementation of the accompanying command-line tools FileJoiner and UnTarBzipper.
-
These tools will be open-source and hosted on SourceForge for the community to freely use, view and modify. Additionally a user guide will be created to guide users through the process of using the tools, and also for setting up a compatible Fedora repository.
Expect this site to be updated with beta versions of the tools within the next 2 weeks.
A set of desktop tools that will make depositing crystallography datasets into a Fedora 2.2.1 repository easier are about to be released in beta on this site. Fedora is the fundamental persistent storage technology behind digital repository software such as Arrow and Fez.
Each tool was programmed in Java for cross-platform compatibility, each having a GUI version as well as command-line functionality. Together they aim to make it easier for researchers to package their data, create valid metadata, both technical and repository-based, upload the data and then unpackage it again once downloaded. The GUI tools are:
RepositoryPackager
Created to take a set of diffraction images, tar archive them, bzip2 compress them and then split the resultant file on a chosen file size. This is done in an effort to make data more space efficient in a repository, and because large files (over 2GB) were found to crash various server software in the upload/storage process.
Uses the apache tar/bzip2 java libraries and incorporates the accompanying command-line tools TarBzipper and FileSplitter.
METSManager
Fedora repositories ingest METS XML files to create and describe entries being stored. This program uses the Harvard METS Java Toolkit to create a fedora-compatible METS package that includes entered values for ingestion. Currently, DublinCore data is created within the METS package to describe basic values such as the title of the object in the repository and the authors. Technical xml metadata relating to the experiment itself is currently put as plain text in the Description field.
Created METS XML files can be validated against the Fedora-compatible METS schema to test their eligibility for ingestion.
Future plans include the embedding of technical XML data such as that described in the data section of this site for data harvesting.
Is a GUI implementation of the accompanying command-line tool METSCreator.
DataDepositor
Once data is organised for upload, and a METS XML file is created, the object and its data are ready to be ingested and uploaded into the repository. This program makes such a process simple by scanning a supplied directory (non-recursive since the structure of repository objects are flat) and uploading each file into a new object described by the METS XML file.
Uses the fedora-management API, and is a GUI implementation of the accompanying command-line tool of the same name.
Currently only compatible with Fedora 2.2.1 repositories.
RepositoryUnPackager
Once a split set of files created by RepositoryPackager are downloaded again from the repository, they can be re-joined, uncompressed and unarchived to restore them to their original form using this program.
Is a GUI implementation of the accompanying command-line tools FileJoiner and UnTarBzipper.
-
These tools will be open-source and hosted on SourceForge for the community to freely use, view and modify. Additionally a user guide will be created to guide users through the process of using the tools, and also for setting up a compatible Fedora repository.
Expect this site to be updated with beta versions of the tools within the next 2 weeks.
Site Update
Some notable bits and pieces have been added to this site, reflecting progress towards the goal of online-hosted datasets that this site will index.
Three More Experiments Hosted
Monash's ARROW library repository site is now hosting three more experiments containing complete datasets. They are linked here on the data page.
Dataset XML Schemas Updated, Example Added
The metadata schemas for diffraction image datasets have been updated - mostly to conform fully to the XML-schema for XML schemas defined by the W3. Additionally an example of experimental/instrument-related metadata descriptions that conform to this schema has been included as a guideline.
Other News
Tools are currently being developed to extract instrument data from image files into TARDIS-compatible XML metadata. Tools are also being developed to compress and split large collections of images in order to be more suitable by internet repositories.
This blog will be refreshed often with development updates.
Three More Experiments Hosted
Monash's ARROW library repository site is now hosting three more experiments containing complete datasets. They are linked here on the data page.
Dataset XML Schemas Updated, Example Added
The metadata schemas for diffraction image datasets have been updated - mostly to conform fully to the XML-schema for XML schemas defined by the W3. Additionally an example of experimental/instrument-related metadata descriptions that conform to this schema has been included as a guideline.
Other News
Tools are currently being developed to extract instrument data from image files into TARDIS-compatible XML metadata. Tools are also being developed to compress and split large collections of images in order to be more suitable by internet repositories.
This blog will be refreshed often with development updates.