Mozilla RDF / Z39.50 Integration Project
Dan Brickley (firstname.lastname@example.org)
Ray Denenberg Z39.50 Maintainance Agency, Library of Congress (email@example.com)
Eliot Christian GILS initiative, (firstname.lastname@example.org)
Sebastian Hammer Index Data (email@example.com)
Chris Waterson (firstname.lastname@example.org)
This is a project to investigate the integration of Z39.50 search capabilities with Mozilla's RDF-based information management environment.
- Make Z39.50 data sources accessible for searching from within Mozilla
- Find an RDF representation of Z39.50 attribute sets
There are thousands of networked Z39.50 servers in existence already. It should be possible to identify some mechanism whereby the Mozilla user interface allows people to send queries to these servers and have the resulting records appear within the standard Mozilla bookmarks/sitemaps interface.
ANSI/NISO Z39.50 is a communications protocol for information retrieval in a client/server environment. Widely used in bibliographic and digital library applications, Z39.50 represents a significant body of experience relating to bibliographic data retrieval and the specification of interoperable semantics for community-specific metadata attribute sets. The Z39.50 Integration Project outlined here is intended to identify appropriate mechanisms that will allow us to exploit Z39.50 data sources and metadata attribute sets from within Mozilla's RDF-based 'Aurora' environment.
This page is intended to provide a useful set of resources from which contributors from the Z39.50, RDF and Mozilla communities can make a start on addressing common problems. Z39.50 can be rather complex for those unfamiliar with the protocol, but there are many developers out there who have considerable experience in this area. What we really need is a good introductory document aimed at outside contributors wanting to hack new functionality into Mozilla in the most sensible way possible. Some of the docs linked from the Mozilla RDF pages make a start in this direction. A good way to get started is simply to download one of the early-release copies of Mozilla and take a look at the way it works.
For Z39.50 developers unfamiliar with Mozilla...
var z3950 = Components.classes["component://z3950service"].createInstance(); z3950 = z3950.QueryInterface(Components.interfaces.nsIRDFDataSource); zapi = z3950.QueryInterface(Components.interfaces.IZ3950Service);
What happens next? Firstly, it is important that people don't go away and expend loads of effort working on something that may not fit in with the bigger picture sketched in the Mozilla development roadmap and on the Mozilla RDF pages. The best way to ensure this doesn't happen is to float ideas on the mozilla-rdf forum (see the Community section for details on accessing this as newsgroup or mailing list).
This should probably be what happens next! It should be relatively simple to put something together (eg. CGI script Z39.50 client outputting an XML/RDF file) which shows what a Mozilla Z39.50 client might look like. Once this is done, it'll be easier to appreciate the issues involved and figure out what we'd want from a full Z39.50 datasource (see below). For example: do we want to be able to store query results as bookmarks? Can we simply use the Dublin Core RDF vocabulary to represent these data structures?
Notealso that the M9 Mozilla milestone includes some non-Z39.50 internet search functionality (based on 'screenscraping' results from HTML pages returned by search engine queries).
Roadmap for a Z39.50 RDF datasource
If native client-side support for Z39.50 is to be integrated into Mozilla, we'll need to find code that implements a Z39.50 client and can be wrappped up as an RDF data source in Mozilla. It looks as is we can use the Yaz toolkit from IndexData, which will give us a pretty solid base for the Z39.50 client code (Yaz ships under a liberal license whose main constraints concern attribution statements and 'no warranty implied'). The main substantive piece of work is likely to be gluing Yaz to the RDF APIs and thinking through the different ways in which the RDF and Z39.50 data models might be related.
This should evolve into a more structured issues list on a separate page; for now, a freeform brainstorm seems more appropriate.
What is Mozilla RDF's notion of a 'searchable thing'? What other networked search protocols will also be supported in Mozilla? Is it feasible to consider adding full Z39.50 support into the client (eg. through contrib'd library?) or will tunnelling over HTTP (via XML/RDF or otherwise) prove to be a more realistic option? What constraints are there on code contributed by 3rd parties? How exactly do we set about hooking up externally authored libraries (eg. Yaz) as XP-COM plugin datasources?
|Issue||Summary||Status and notes|
|Yaz library||Can we get confirmation that Yaz license can be used with Mozilla?||Need advise from Mozilla org; This appears to be a Frequently asked question. It looks like we don't have a problem here...|
|Service description||We'll need an XML/RDF representation of the properties of various Z39.50 services (connection details, subject coverage etc)||The GILS Locator Records (perhaps munged into RDF and Dublin Core?) seem a good basis for this. Important to coordinate with descriptions of SmartBrowsing and other RDF data services in Mozilla. Need to find out how this works in rest of Mozilla (eg. localised configuration of cache services, use of CSS/XUL for icon sets, chrome etc...). Also relates to RDF Sitemaps: a sitemap for some Web service should probably be allowed to contain RDF/XML describing Z39.50 search targets (and other machine interfaces) where available (need docs on RDF Sitemaps).||Where does Necko fit in?||Necko is the new Mozilla networking library. Connections to networked servers should go via Necko. It's not immediately clear how to 'do the right thing' here.||The Necko overview helps a little but we really could do with some example code to copy... Ideally we set about writing a protocol handler associated with Z39.50 URLs. We also have the notion of Z39.50 tunnelled over HTTP to bear in mind, which might crosscut such developments...|
|Non-Z39.50 searching||There are likely to be non-Z39.50 search services accessible via RDF/XUL, eg. portal searches, 'find' in bookmarks. Important to avoid needless duplication, and to get a roadmap for such work from Mozilla organisation to plan/coordinate etc.||Bug 11347 against M9 has been filed as a reminder that the roadmaps need updating. Best way to avoid duplication is to keep discussion on the public forums, and read back through past threads on RDF search in the newsgroup.|
|BER utilities||Z39.50 uses ASN.1 and BER encodings; Yaz provides code for dealing with BER. However, Mozilla's LDAP library will also have BER tools. What best to do?||
Find out whether the LDAP BER utilities be explosed via XP-COM or
otherwise usable. Does the duplication matter that much - versus added
complexity of liaising with LDAP work. Aside: is an LDAP RDF datasource
From Sebastian: If we can build it all as a run-time loadable module, then the overhead of carrying two BER stacks around is less significant. I believe the size of the BER code itself is smaller than the higher-level layers which encode the actual, structured Z39.50 (or LDAP) PDUs.
General consensus that is better to use the Yaz BER libraries
See the main Mozilla RDF pages for links to relevant lists/newsgroups.
- Z39.50 Maintainance Agency at the Library of Congress
- Z39.50 Core Services - a nice overview of the protocol
- The Z39.50 Information Retrieval Standard (D-Lib article by Clifford Lynch)
- The Z39.50 specification (the spec itself)
- Example datasources - GILS Locator Records describing in XML a large number of Z39.50 servers
- a draft proposal from Eliot Christian that provides an RDF Schema for the GILS locator records (see also an opinion piece on Z39.50 and the RDF Semantic Web...)
- The Yaz Z39.50 programmer's toolkit from IndexData (candidate code for use with Mozilla)
- Profile for Z39.50 over HTTP (draft)
- Z39.50 Attribute Architecture
- The XER encoding rules (proposal to replace the traditional BER encoding of ASN.1 data structures with an XML representation)
- More Z39.50 related resources for background reading.
- See also Query Language Issues in a Distributed Indexing Environment (discussion paper from QL'98 workshop)
- Mozilla development roadmap
- Getting involved...
- RDF Datasource HOWTO
- XP-COM home page
- Roadmap for XPCOM, XP-Connect, XPTCall and XPIDL
- it might be worth having a look at the directory / LDAP code in Mozilla, since the LDAP libraries provide tools to handle BER encoded messages.
This is an exploratory project, and an experiment to see whether we can make it easy for the Z39.50 community to contribute their expertise to the search environment under construction for Mozilla. It does not constitute any commitment to ship Z code with the '5.0' release, nor a promise that a great deal of engineering resources can be poured into this problem. This probably doesn't need stating, since Mozilla isn't NSCP/AOL. Basically, it's likely that some XML/RDF-based client/server metadata search functionality will appear in the Mozilla browser. Hopefully, this will be informed by and compatible with existing Z39.50 implementations. But we need your help...
Author: Dan Brickley email@example.com
Last Updated: $Id: z3950.html,v 1.7 2003/02/21 19:38:59 timeless%mozdev.org Exp $