Document RepositoriesDuring the early 1980s, I spent six months prototyping a retail cash-processing application in Applesoft BASIC, and another six months with a compiler and a 6502 assembler honing the software to perfection. One day my boss called me into his office. "Look at this," he said, flicking through the menus and the all-too-familiar screens. "It's so simple." I flushed with pride; here was recognition at last. "Yes," I agreed. "It is simple." "So what the hell took you a year?," was the response.
Lotus Notes is a little bit like that: It has been evolving for nearly as long as Microsoft Windows has, and you can only guess how much was thrown away over the years. But what's remaining is deceptively simple.
Deceptively? Yes, because from an IT (information technology) perspective, you can suddenly build bigger and more distributed systems--with all the problems that process implies. It's also deceptive because such systems typically require conventional RDBMS (relational DBMS) capabilities that Notes awkwardly lacks. But for end users swamped with piles of documents, Notes makes it easy to build real client/server applications to help manage those documents.
Documents, Views, and Forms
A Lotus Notes database contains only documents--its data records. But these data records are miles ahead of the conventional kind (e.g., "Given x bytes per record, record N must be at offset N x."). So, designer Iris Associates (which is now a part of Lotus Development) coined the term document for a dynamic data record of indeterminate size that can store any of seven different types of data: text, text lists, numbers, number ranges, times/dates, time/date ranges, and type composites.
A type composite, also known as a rich-text field, is a flexible, self-describing format that can store just about anything that you care to paste or embed into it. This includes word processor files, spreadsheets, bit maps, OLE objects, sounds, hypertext links (i.e., doclinks) to documents in the same or a different database, and even--if a C programmer using the Notes API so arranges--user-defined data types. To avoid confusion between a document in the Notes sense and a conventional document (e.g., a word processor or spreadsheet file), I will hereafter refer to Notes data records simply as notes.
Views are windows for browsing through the database; they're sorted by pertinent data fields. Each column in a view either maps to a field in each note or shows the result of a Lotus 1-2-3-like formula. Using categories derived from the values in a database, a view can expand and contract in much the same manner as the outliner in a word processor does.
Forms are templates for entering and displaying a note's data. A form can contain static text, bit maps, data fields, and command buttons. The fields of a form map by name to the fields of a note, so a user can enter data with one form and view it using another.
From a user's perspective, you double-click on a Notes database icon to open a window containing a view. You then scroll through the view to the note that you want and double-click on it; another window opens to display the note's data in a form. It might contain, for example, an Ami Pro icon. Double-clicking on that launches the application and opens an embedded Ami Pro file for editing. When you quit Ami Pro, the file is stored safely back in the note.
Because notes are such flexible containers, a database can store all the elements of its own user interface within itself--icons, forms, views, macros, and so forth. Thus, each database becomes a completely self-contained application.
Transmitting Documents by Replication
Replication means cloning a Lotus Notes application (or database--the terms are usually synonymous). There's no restriction on how many replicas can exist, and no single replica serves as the master. Each replica accumulates its own unique additions and deletions (the latter are represented by deletion stubs).
Replicas synchronize in two ways. Server-based replication is typically a scheduled affair; a task on one Notes server opens a dial-up or LAN connection to another. Client-based replication occurs intermittently at a user's discretion--perhaps from a laptop that dials in to a Notes server from a hotel room.
Either way, the involved parties first establish each other's identity. Next, they consult their replication histories and build a list of the data notes, design notes, and ACL (access-control list) notes that have changed since the pair last replicated. Subject to each application's access permissions from the other, they then start to exchange notes. The first note to be swapped is the ACL, which can then govern access permissions for the rest of the exchange.
After all notes have been successfully swapped, both replicas update their replication-history records. Since an application doesn't know which other applications it will replicate with in the future, or when, all deletion stubs are kept intact. They're finally removed after the deletion-stub-purge interval, which is typically 90 days.
What happens when, say, my coworker Fred and I use different replicas of the same database, which contains a note that has an embedded Ami Pro memo? I make some edits. Fred makes some different ones. Then we replicate with each other. (In a real-life situation, we'd likely both dial in and replicate with the office server, but the effect is the same.)
Sooner or later, however, a fight will inevitably develop. In fully connected client/server scenarios, user-interface software arbitrates overlapping edit requests. But Notes, only intermittently connected, handles conflicts the only way it can. Both notes end up in both databases, and the one that loses the editing battle is marked "Edit or replication conflict." From then on, it's up to the humans to resolve things.
The Notes Document Repository
Some say that in any organization with more than 500 employees, there are no medium-size Notes databases. (And with its current average of 230 users per site, Notes isn't selling to small enterprises--yet.) A typical large organization has either one big application, with professional developers, thousands of users, and a budget to match, or a little database used by a team of about eight people, built by an IT-aware member of that team.
Typically, a large organization will start out with one big Notes application, often a personnel-oriented one with a hybrid client/server architecture. Such an application enables smaller workgroups, typically driven by the need to manage documents, to build and use their own Notes applets.
All organizations, large and small alike, have a standard document repository: the file server, a labyrinth of directories with thousands of files named in the DOS 8.3 format. Users can more effectively classify, navigate, and search for those files if they are placed under Notes' control. To create a simple Notes repository from the standard Document Library template, you only need about the same level of skill required to produce a useful spreadsheet.
Link, Attach, or Embed?
Word processor files can be left in native DOS format and linked to a note. Alternatively, there are three ways to move the contents of files directly into Notes--linking, attaching, and embedding.
Based on Microsoft's DDE protocol, linking is a kind of dynamic copy-and-paste operation that's just too clunky to be of any real value within an occasionally connected workgroup. I've seen grown men break down and cry trying to get the Paste/Special/Link menu option to work smoothly and consistently across a network. And if you're trying to work at home away from the LAN, forget it. That said, however, the two most useful protocols, DIP (Document Insert Protocol) and OLE, are both layered on top of DDE.
Any kind of disk file can be attached to a note. Notes stores a copy of the file in the document database, representing it as an icon in a rich-text field. DIP works by looking in the NOTES.INI file for the DDE strings required to launch an application and to pass it a command to open a file. The application need only be capable of being a DDE server. "DIP will probably go away by Notes version 3," a Lotus spokesperson once commented, "by which time most applications will be supporting OLE."
But today, on the eve of Notes 4.0, DIP is still with us and should not be underestimated as an effective tool in the document-repository designer's kit. Why? It has the merit of simplicity. When you double-click on a file-attachment icon, Notes asks whether you want to detach the attachment (i.e., extract a copy as a separate file) or launch it. If you choose to launch it, Notes creates a temporary disk file and then passes its name to the host application.
This technique works well when most material is read-only or is created by a few and read by the majority. But updating a file in place is awkward. You have to save it by another name, switch to Notes, put the note in edit mode, delete the existing attachment, attach the new file, save the note, and remember to delete the saved file from your disk.
By contrast, a document in the form of an embedded OLE 1.0 object is a lot easier to update. Like the attachment, a word processor file resides physically in the database. And, as with an attachment, the host icon is displayed in a rich-text field. But double-click on the icon, and you land in a new host application window with the file loaded (see the screen on page 132). Edit away, close the host application window, and the edited file is written back into the note. Since this is an in-memory operation, there's no temporary file to complicate matters. Lotus Notes 3.x supports OLE 1.0 as a client, not as a server.
For Unix users, there's LEL (Link, Embed, and Launch-to-Edit). Lotus Development and Software Pundits jointly designed and produced LEL at the same time that Notes was ported to Unix. LEL is the functional equivalent to OLE 1.0 and is specially designed to let users store and retrieve documents in a mixed Windows/Unix environment. When an application is available on both these platforms, objects that are created on one platform can be launched and run by the application on the other platform.
At the time of this writing, only Ami Pro 3.0 enjoyed this exalted position. Like Microsoft, with its OLE toolkit, Lotus and Software Pundits provide an LEL toolkit to enable Unix developers to incorporate the technology, both client and server, in their applications.
S. G. Warburg's Stream Repository
File attachments are a central part of an international marketing system, called Stream Repository, developed by the corporate finance division of investment bank S. G. Warburg. Jeremy Stone, the director responsible for the project, remembers how it got started. "Our worldwide marketing effort, much of it [involving] pitching to governments for the job of privatizing huge public-sector companies, relied heavily on hand-crafted, color-printed books presented to the people who made the decisions," he recalls.
"Yet all those books had to be produced by one or two centralized desktop publishing departments. The business is getting more marketing-intensive, and also turnaround times were getting shorter, creating a potential bottleneck. We decided that rather than increase the size of the central units to cope with increased volumes, we should take the opportunity to integrate the process of document production with all the other things that have to be done by corporate finance executives preparing for a transaction," he continues.
"We had to take the PC-based office software as given--Word and Excel were the group standards. Attaching key Word or Excel files to Notes forms enables us to control content and style and avoid [the need for] people reinventing wheels every day. The main components of the document repository are standard, frequently used items that can be freely mixed and matched in different documents but individually need to be controlled and kept up to date. The Notes database enables us to do this and also leverage the power of the tools we started off with," he concludes.
The principle behind Stream Repository is that a complete, street-ready presentation document can be assembled in Word with the aid of templates and macros. Graphs and diagrams are produced in Excel. A custom Chart Wizard, written in an Excel macro, allows the user to create graphs and place them on the clipboard to be pasted into the Word document. Lotus Notes databases are used at the beginning as a repository of frequently used, ready-formatted Excel charts, logos, and Word text paragraphs, and again at the end to store the completed presentation documents.
The simplicity of file attachment has worked well for the Stream Repository system. It has reduced potential problems in a combined Mac and PC environment as well as reducing support costs as the new technology has been rolled out worldwide.
Because the file attachments can't be indexed and searched by Notes' Verity Topic engine, the note that carries the attachment icon also has over a dozen different simple-text fields (e.g., Country, Sector, and Type) that help users find what they're looking for as they navigate views of the database. In many cases, the note also carries keywords and even a short abstract to support fulltext search.
Coopers & Lybrand's CLASS
CLASS (Coopers & Lybrand Audit Support System), developed by the accounting firm Coopers & Lybrand, uses embedded objects. This system is a powerful implementation of Lotus Notes as a document repository.
As Rod Parry, the U.K. international partner for CLASS development and rollout, explains, "As you would expect, within C&L, the audit process is well understood, and it is, by definition, a workgroup activity. On a complex international audit, team communication is paramount, with perhaps 30 auditors in the U.S. and several groups of 10 in different locations around the globe.
"All of the team must have all of the audit file, all of the time. In addition, the system must enshrine our principle of TEQ [Total Engagement Quality] for each and every audit."
The CLASS system is already being used by 4000 people in 18 countries, with a planned 15,000 users in 25 countries by the end of 1995. It's split into three modules: Strategy, Audit File, and Audit Financials.
For each audit, the Strategy module, which is written in C, is used first. After an in-depth parameter-setting session, the Strategy module builds and populates a new Lotus Notes application, the Audit File, which is used as an audit plan, control-documentation store, on-line issue clearinghouse, and document repository.
Ami Pro and Lotus 1-2-3 are the word processor and spreadsheet used for the CLASS system. The templates for Ami Pro files and 1-2-3 spreadsheets are kept in a separate Lotus Notes database, called the Template Manager. This makes replication around the globe easier and ensures that all template files are up to date and fully meet legal requirements. Within the Template Manager, Ami Pro documents are embedded in a rich-text field, and a button on the form lets users copy the object to the clipboard.
When users switch back to the Audit File module with the insertion point in the required field, the Edit/Paste/Special/Embed menu option embeds the Ami Pro file in the note. As the audit proceeds, a complete database is built up, containing not only electronic copies of all letters, memos, and spreadsheets, but also all the audit issues that are signed off as dealt with and all tasks that are signed off as done. Consistent quality is ensured for all audits, from one-person jobs to the largest international engagements.
Fulltext Searches
In practice, fulltext searches within a Notes document repository are rare. There are two reasons for this. First, attachments and embedded objects cannot be opened by the search if they are represented on a note by icons, although rich text pasted into a field is searchable. (This restriction is lifted in Notes 4.0, which can index attached and embedded content.) So, unless the host note carries a pertinent text abstract in a plain-text field alongside the object icon, it's invisible to the indexer and the search engine.
Second, a note usually carries other text fields (a letter, for instance, would have To, From, Title, and Date fields). Thus, it requires less thought, though a little more time, to find "that letter I sent to Jones last week about office equipment" in a view than it does to master even the simplest arcana of the fulltext search bar.
FX
FX, short for field exchange, is a valuable feature for document repositories. It allows the bidirectional exchange of data between a document, which is an OLE 1.0 object, and text fields in the note to which it belongs. Supported applications include Ami Pro 3.01, Freelance 2.01, Improv 2.1, Lotus 1-2-3 release 4.01, and, most recently, Microsoft Word 6.0c and Excel 5.0c (later versions of all these programs are also supported).
FX requires an OLE-embedded object in a note, SHARE.EXE (or VSHARE.386) to be loaded, and one or more hooks from the note to the object. In Ami Pro, for example, these hooks take the form of special bookmarks in the file (e.g., @TestField) and identical field names in the note (e.g., TestField). Note the leading @ for the bookmark name in Ami Pro.
When the object is launched, all fields that have matching names are read from the note and are used to update Ami Pro immediately before it displays the document. In a similar manner, when the host application window is closed, all the bookmark names are read and used to update the note.
With FX, for instance, the names of the fields in the note and their corresponding bookmarks in Ami Pro might be something like ToName, Organization, Address, DateOnLetter, Subject, and Title. If these bookmarks are edited while Ami Pro is running, the edits automatically update the fields in the note when the document is closed.
As mentioned earlier, a good Notes document repository is based on information stored in the note alongside the object. FX makes this job much easier. As you might expect, the technology used for FX is based on OLE; the object application is the server, and Lotus Notes is the client.
Further Refinements
It's possible to build automatic object activation into the design of a note so that when you open the note, the contained object launches. You can also control which actions--for instance, creating, reading, or editing of the note--invoke this behavior. Further, the note itself can be displayed or hidden.
When combined with FX, this produces the purest form of document repository. You open the Notes application, navigate a view to find the title that you want, and double-click on it. Your word processing application is then launched with the document in it. If you edit the title and close the document, the view will reflect the new title.
Because Notes lacks a distributed locking mechanism, it isn't a document manager in the purest sense. However, it provides a useful versioning mechanism. If you edit an embedded object within a note (or just edit a note), Notes saves a new note instead of overwriting the existing one, building a link between the two. This link ensures that, even if users recatego-rize on the fly, all the versions of a note are always clustered together.
Lotus Notes can support enterprise-document repositories, but it's also useful at the workgroup level. Users can store documents where they can easily find them again, use long names, categorize on any combination of fields, and painlessly replicate across continents, as well as to and from laptops.
This combination of features is certainly compelling. Lotus understands that Notes doesn't have to be the most powerful document manager or have the ability to satisfy all the criteria of, say, a professional librarian--it just has to be easy to use. And that it is.
During the edit of an embedded file, double-clicking on the icon in the rich-text field of a note launches the host application under OLE 1.0 control.
Copyright © 1994-1995