Link Associated Computation in HyperNet

NENAD MAROVAC*

San Diego State University, San Diego, CA 92182, USA

HyperNet is an authoring and browsing system for the creation and navigation of multimedia documents. It was designed to provide a mechanism for very fast choreographing and configuration of hypermedia documents of arbitrary structure and content. HyperNet features enable a group of users scattered world­ wide to compose hypermedia documents from components which exist in a set of world-wide distributed data bases. In this paper we present two new features in HyperNet. The first feature, synchronization clauses in HyperNet links, allows for synchronization of multimedia components in hypermedia docu­ ments. The second feature, active links, is not commonly found in hypertext machines. These links associate with them extensive computations. Active links add new functionality to HyperNet which includes interface to and browsing of external distributed data bases and interfaces to and execution on demand of computational modules. This new functionality extends the use of HyperNet to new application domains, e.g. Decision Support Systems, Group Support in Cooperative Work, Interactive Database Browsing, Software Development Environments, and Interactive Linking and On Demand Execution of

World-Wide Distributed Reusable Modules.

HyperNet is an authoring and browsing system for the creation and navigation of multimedia documents. It was designed to provide a mechanism for very fast choreographing and configuration of hypermedia docu­ ments of arbitrary structure and content. HyperNet features enable a group of users scattered world-wide to compose hypermedia documents from components which exist in a set of world-wide distributed data bases. This capability may be needed when dispersed profes­ sionals are collaborating on the creation or modification of a document. It provides an environment for users in which they would be able to browse, search for and retrieve hypermedia documents or their components, whether they and/or the documents were local or remote, with the actual retrieval process being transparent to the users. HyperNet also supports group cooperative preparation and browsing of multimedia documents.

A hypertext system is composed of nodes and links (Nielson, 1990). Nodes store the information and links represent the associations between the information con­ tained in these nodes. The presentation (display) format for nodes, is called frames or windows depending on their size and they need to be scrolled to be 'read' completely. In this paper we use the term object inter­ changeably with nodes to emphasize the object directed processing of nodes in order to generate their presenta­ tion (reader-oriented format) format, as well as to stress the multimedia nature of the nodes (text, graphics, audio and moving video frames). The formatted presentation form of nodes we will call frames (even though HyperNet

* At the time of completion this document the author was a visiting scientist with the FAW-Germany's Research Institute for Applied Knowledge Processing, VIm, Germany.

text and picture presentation (display) format for nodes can be larger than the screen and are scrollable) again to emphasize that the presentation format of nodes contain frames of text, pictures, audio and moving video.

Frames in a hypertext system are either of a static or a dynamic nature. The static frames are pre-generated during document production, i.e. they are stored in display format, and do not require any additional pro­ cessing to be displayed on the screen. The dynamic frames are stored in an encoded, usually compact format, and they require computational processing to generate a display format in order to be displayed (Nielson, 1990). This processing takes place when the user clicks on a hotword (or a more general anchor) in a source frame. Since this action and processing is related t6 the traversing of the link associated with the anchor, we will refer to this computational processing as link related formatting computation, and the the activity of processing the destination node to produce its presentation (display) format we will refer as elaboration of the node. The dynamic frames manifest the effort to store the content of frames in a more compact form and sacrifice the navigational time response by requiring computational processing and, therefore, creating some delay before the frame can be displayed. Typically, such delays are very small and unnoticeable by the user. However, the dynamic frames in addition to the compactness also offer more flexibility as the format in which they are stored may be high level and allow for editing of frames by users. Usually, within one system only one encoding format is used for all frames and the computational processing is a transformation of information from one (the encoding) format into another (the display) format.

HyperNet uses dynamic frames. However, the compu­ tational processing in HyperNet depends on the class of

the corresponding node, i.e. HyperNet uses the following model for link related computation:

presentation-format (destination node)

= p [destination object class ] (destination node content) where the square brackets are used to indicate that the processing function itself depends on the class of the destination object.

Classes which HyperNet supports are described in Marovac and Osburn (1992) and Marovac (1993).

It is very important to identify and stress the uni­ directionality of the link related processing, i.e. when the user clicks on an anchor, the associated link is followed to identify the destination object, its is class recognized, and the link related computation is invoked to generate the presentation format for the destination node and the result is activated (displayed or played) as the destination frame. [At the risk of being overly verbose to describe something known to all hypertext affectionados, it is important to clearly identify this sequence of actions, as the process sequence that is described in this paper (e.g. active links) and incorporated in the recent release of HyperNet (HyperNet 2.0) differs considerably from this standard one.] The one-directionality of the process also means that this link associated processing only affects the destination node object, i.e. has no effect whatsoever on the source node.

In the recent release of HyperNet (HyperNet 2.0), a considerable new functionality has been added. For some time we felt the need to add to HyperNet a mechanism to support synchronization of document components of different media. More recently, we also decided to expand HyperNet and its language Scripts to support other application domains. This paper presents the results of this effort and introduces the HyperNet lan­ guage constructs supporting the new application domains. Some of the applications considered are:

• Decision Support Systems.
• Group Support in Cooperative Work.
• Interactive Database Browsing.
• Software Development Environments.
• Interactive Linking and On Demand Execution of World-Wide Distributed Reusable Modules

In the HyperNet version 1.0, a link is defined as:

LINK (title = object-name, hotword = a sequence of words)

If an author wishes to introduce a link to a document component in his document, he must specify only the name of the target document component and the desired hotword. He needs not be concerned with the class of the object-name or the link related formatting computations.

When a reader clicks on the anchor depicting the link to the object-name, after accessing the object-name from the world wide distributed data base, the system checks the object-name's class and performs the required com­ putational processing.

In HyperNet 2.0, the link definition and the link associated processing is considerably expanded in order to provide support for:

1. Synchronization of multimedia components.
2. Interface to and browsing of external data bases.
3. Interface to and execution on demand of computa­ tional modules.

In the remainder of the paper, we discuss these added features and the associated functionality in detail. We also address benefits provided by this new functionality in new application domains. However, we do not go into details about motivations and the role of HyperNet in these applications. This is covered in other publica­ tions, e.g. Wynne et a!. (1992), Wynne and Marovac (1993) and Marovac (1992a).

In the next section, Section 2, we discuss modifications to the link structure in HyperNet in order to introduce features needed to synchronize document components of different media.

Synchronization of multimedia document components of different media was studied by other researchers (Little and Ghafoor, 1990; Steinmets, 1990; Sato et a!., 1993). What differentiates our approach are two rather unique objectives that we had in mind during the conception and design of HyperNet 2.0:

1. Open system architecture. The system has to support any major industry standard in encoding of informa­ tion of different media, i.e. if and when a new standard becomes available and takes an important role in the multimedia 'universe' we should be able to 'plug it in' without a significant incremental modification in design and coding.
2. World-wide distribution of the system and document components. Our system provides for distribution of the system, users and documents over a very large geographical areas. Therefore, the synchronization primitives and the associated mechanism must be able to handle the fact that the 'initial' fetch of components of different media could be and usually is from large distances.

Sections 3-5 discuss a novel concept, the active links, i.e. links which have computational activities associated with them. They may be used to propagate computation and information flow from source nodes to target nodes, vice versa or both. This enables them to function as interfaces to external environments such as data bases and computational systems.

2. HYPERNET LINKS

In the HyperNet version 2.0, the definition of a link has been extended. In fact, we have two types of links:

1. Passive links.
2. Active links.

i.e. in HyperNet 2.0 we formally define links as:

The passive links do not effect any changes in the information content in either source or destination nodes, i.e. t: only computation associated with these links is link-related formatting processing of destination nodes in destination frames.

The active links on the other hands effect the changes in the information content or environment in either sour nodes, destination nodes or both.

The definition of a passive link is similar to its definition in HyperNet 1.0. The major change in the definition is tJ introduction of synchronization options to support simultaneous, sequential and cascading activation of frames different media. The new definition for a passive link is:

<passive-link -statement) <hotword -specification) < object-expression)

: = LINK, <hotword-specification), <object-expression) : = hotword = a-sequence-of-words

: = «object-specification»)

[<object-operator) «object-specification»)] + 1

< object -specification)

: = <delay) title = object-name <duration) :=*1+1,1->

: = duration fixed-point-expressin, I NULL : = delay fixed-point-expression, I NULL

< object-specification) <object-operator) <duration)

<delay)

where the pair of square brackets [] followed by the plus' +' sign in the statement defining 'object-expression' a used to indicate one or more occurrences of the expression within the brackets.

For example, the LINK statement

LINK, hotword = hotwordl, (title = componentl) object-operator! (title-component2)

specifies an object group. It indicates link to a group and a group activation. When a reader clicks on hotwordl I

,,,actually initiates activation of all objects in the group. This is very useful when a document author wishes to combil together, e.g. a picture with a text description of that picture and an accompanying audio commentary. Using su( statement, he indicates that activation and presentation of all three components should be triggered by one click ( the associated anchor.

The object-operators are defined as:

1. * This operator is used to indicate that two or more components are to be started at the same time. An examp of the use of such expression is

(title = picture) * (title = audio-description)

This example indicates the intention to display a picture and to play the audio commentary for the picture at t1 same time. If any of the media document components are not present locally and have to be fetched from a remo site, the use of the * operator indicates that the system has to wait until all components are present and then 2 components are started 'simultaneously' beginning from the left side to the right side of the expression. In othl words, the author is indicating that it does not make sense to start either the picture without the narrative or tl narrative without the picture.

2. + This operator is very similar to the '*' operator. The only difference is that if the component on the left side I the expression is available locally and the one on the right side is not, the one on the left side may be started c its own and the one on the right side is started as soon as it arrives. Such scenario could take place if the documel author again combines a picture with a commentary. However, in this case he feels that the picture could 1 enjoyed on its own even if the narrative is not immediately present. The picture will be displayed so that the readc may start 'to visually enjoy it' and the narrative will be played as soon as it available.
3. , This operator denotes a sequential concatenation of components. The next component is started when t1 previous one has completed. An example of the use of this operator is when the document author wishes 1 indicate that a set of components should be sequentially played one after another. However, the entire sequen(

in initiated by one click on a hotword. (One should note here that some document components have a natural completion time, e.g. an audio ar a video clip. However, other document components may not have a natural termination time, e.g. a text or a picture frame. In such cases, the completion time is specified by the author via the duration clause to indicate the desired length of the presentation for the component. The reader can override the author selected duration by clicking on the component frame to terminate its presentation.)

-- This operator is used to indicate a cascading relationship between components. The expression 'a--b' is used to indicate that the component b is to start a desired time interval after the start of the component a. The desired time delay between the two starts is specified using the delay clause, e.g. the statement

component-a--delay 0.50 component-b

ecifies that the component-b starts 0.50 s after the start of the component-a.

HyperNet version 2.0, three types of active links are introduced. Computational processing is involved with all pes of active links. This computation is called active link computation, or, for short, active computation. The active mputation is in addition to the link related formatting computation.

An active link is defined as:

(active-Jink-statement): = (UPLINK-statement) I (DNLINK-statement) I

(BILINK-statement)

le first type of active links, called DNLINK, projects the results of the active computation towards the destination Ide. The second type of active links, called UP LINK, projects the results of the active computation backwards from e destination node towards the source node. Finally, the third type of active link, called BILIN K, is a combination DNLINK and UPLINK and projects the computational processing both from the source node to the destination ,de, and from the destination node to the source node. However, the bi-directional computation effects do not take ice simultaneously. The DNLINK type of computational processing takes place first, then the destination node is lborated, and then the UPLINK computational processing takes place.

We will discuss in detail all three types of the active links, starting with UPLINK.

1 UPLINK active link may have more than one destination object, i.e. it may link a source node to one or more stination nodes. The active computation associated with an UPLINK link affects the source node containing the char associated with the UPLINK. These effects are implicit in the sense that they occur spontaneously during the lboration of the source node, i.e. before the reader has clicked on the corresponding anchor. These effects will, ·wever, depend directly on the current information content of the destination nodes associated to the source node rough the UPLINK link.

An UPLINK is defined as:

~UP-object -schema) ~UP-link-schema)

: = UPLINK, (hotward-specification), (object-specification),

(UP-object -schema), (UP-Jink -schema)

: =hotwordClass = (hotward-class) I hotword = hotword-name : = integer I float I fixed I record I image I .',

: = (objects) I (object-classes)

: = title = object-name I

(title=object-name [, title = object-name] *) : = objectType = object-class-name I

(objectType =object-class-name [,

objectType = object -class-name] *)

: = UPobjectSchema = object-schema-name : = UPJinkSchema = Jink-schema-name

lere the pair of square brackets [] followed by the sign asterisk '*' in the statement defining 'object expression' are ~d to indicate zero or more occurrences of the expression within the brackets.

This group of statements should be interpreted as follows:

Object specification identifies one or more destination objects, i.e. the destination nodes.

The computations specified in the UP-abject-schema are applied to these objects. The result is a logical frame. The computations specified in the UP-link-schema are applied to this logical frame to produce a user active view. This active view becomes a segment within the source frame.

The active view can be either:

1. a passive visual segment in the source frame
2. an audio or a video clip which is activated when the source frame is elaborated (completed)
3. an active visual object in the form of: - text: alphanumeric or numeric

- a structured computational object, e.g. an array or a record

- a structured picture, or

- an image

When the active view is an active visual object (case 3), it then also plays a role of the anchor (hotword) for the link. For the active frame to be displayed as an visual object, it is still necessary to perform the link related formatting processing to transform the active view into its display format within the source frame. This processing depends on the object class of the hotword.

We should note the alternatives in the specification of the hotword, i.e. the hotword-name and the hotword class.

When the hotword is specified by a clause hotword = hotword-name the implications are that:

• We have a hotword in the normal sense of a hypertext document.
• The active view is a passive segment in the source frame.

However, when we specify the link hotword through a clause hotwordClass = hotword-class, then the implications are that:

• The active view is an active segment in the source frame.
• The actual hotword for the link is unknown.

At the time of planning the source node (frame), its author knows the class the hotword will have but he does not know the actual 'value' that the action view, i.e. the hotword, will have at the time of the source node elaboration. The value for the hotword is actually determined at the time of the source node elaboration. It is produced in the link associated computations specified in the object and link schemas. The actual value of the hotword will depend on the information content of the destination nodes at the time of elaboration of the source node. That value will be different at different times as readers arrive to the source node. In fact, the value of the hotword may also change while a user is 'reading' the source frame if the content of the destination nodes within the HyperNet data base are changed by an external application. One should also note that the value need not be a single number or a text string, but a structured graphics or an image.

We note two alternatives in the object specification. The first alternative, objects, is used when the destination nodes participating in the link are known at the time of preparation of the HyperNet document, and they are unique. Therefore, they can be directly specified in the link statement.

The second alternative, object-classes, is used when the object class for the destination nodes is known, but it is not possible to identify the destination nodes themselves at the time of the document preparation. This may be for a number of reasons:

• They are unknown at the preparation time.
• They may differ with time, even with each visit to the source node containing the link.
• There may be a large number of objects that can and are used as the destination nodes either:

1. Selectively, one at each visit to the source node. In this case, the system gives the reader a prompt for the name of the actual destination node within the class. This prompt is issued at the time of elaboration of the source node.
2. Cumulatively, i.e. all together contributing to the generation of the active view.
3. Iteratively, in a sequence one at the time allowing the reader to visit all of them if he desires.

They actual names for the object(s) can be supplied by the reader of the document at the system prompt. This prompt will be given to the user as the source frame is being generated.

Finally, we should address the two computational elements in the link definition, the object-schema and the link schema. (Although, in the actual definition of UPLINK, DNLINK and BILINK we use the prefixes UP, DN and BI, e.g. UP-object-schema and UP-link-schema, we will omit these prefixes when discussing the concepts behind the schemas.) They are used to specify details for the link associated computations which extract data from the destination nodes and process that data to produce the active view within the source frame. The specification for link associated computations include:

• Identification of objects, i.e. the destination nodes. This identification can contain clauses that specify cumulative, iterative and selective processing of all objects satisfying certain conditions, e.g. expression in relational calculus or SQL.
• Identification of necessary data within the desired destination nodes.
• Specification of algorithms for the calculations.

The processing specified in the object-schema produces a logical format for the active view. This is similar to the processing required to extract and transform data in a number of records on the physical level of a data base in order to produce a logical record on the data level in the data base (Date, 1990).

The processing specified in the link-schema is applied on the logical format of the active view in order to produce the actual user format of the active view. This processing is similar to the one used to transform information in a logical record on the data level of a data base in order to generate user view record(s) in an application (Date, 1990).

The idea behind the UPLINK computations and generation of active views is similar to the one found in form generation and tax preparation packages. There, the information within forms on the lower level is processed in order to produce an entry within a form on the logically higher level. However, in HyperNet this processing can be linked to information in external data bases and produces numerical entries, complex computational structures, structured pictures, and audio and video excerpts.

Why did we separate the sequence of computations into object and logical schemas? Often two logically separate computations are required to produce an action view. For example, in producing a segment view from a picture of an engineering assembly, the role of object-schema would be to collect all parts of the assembly and build a model for the assembly. The computations in the link-schema would perform graphics related transformations, like rotate the assembly, project the three-dimensional model of the assembly into a two-dimensional projection and then clip the two-dimensional picture to obtain the desired segment as the active view. Partition of the computations into two schemas also allows for simpler generation of separate two-dimensional segments showing different parts of the same engineering assembly when these two-dimensional segments are all based on the same three-dimensional model of the assembly generated by computations specified in the object schema.

The object and link schemas may be in many different forms, e.g. a specification of a sequence of procedures to be executed, operating system shell scripts, relational algebra expressions, or a form of meta language specification for required algorithms.

There is also a third segment of computation to actually produce the user (the presentation) format of the active view. This computation, specified implicitly by the class of the user view and the hotword class, is the processing required to transform the user view class of data into the hotword class of data (the actual presentation format).

5.1. Application related motivations for UPLINK

Introduction of UPLINK modes of links into HyperNet expands the use of HyperNet into a whole new areas of applications. Here we will try to touch some of them:

• Dynamic browsing of data bases. HyperNet presents a user friendly way to navigate through external data bases.

In this class of applications one can feel the benefits from the separation of the necessary link associated computations into two schemas. The object schemas allow us to specify and effect flexible interfaces to different style data bases. The distributed aspect of the HyperNet access mechanism to document data bases which are distributed world-wide (Marovac and Osburn, 1992), also enables us to use HyperNet in browsing distributed data bases provided they are connected to Internet sites. The link-schemas allow us to specify and effect user oriented transformations of the data accessed in the external data bases.

• Decision Support Systems. Extraction and condensation of information from the lower levels of abstraction and presentation of the results on a higher level of abstraction represents an useful tool in rapid decision making situations.
• Picture and image processing. Object and link schemas allow for demand generation of three-dimensional structured models and extraction of information required on demand in order to present pictures of segments in the form more appropriate to a given situation than static pictures in the form stored in a multimedia systems without active links.

These links are also used to associate active computations with a link. The active computations specified in a DNLINK affect the destination nodes only. These effects are explicit in the sense that they occur after a user clicks on the hotword representing a link from a source node to the destination nodes. However, the information from the source node is used in these computations. These computations may produce data. That data is then used to elaborate the destination nodes to produce the associated destination frames, and it may also be passed to executable components of destination nodes.

The definition of a DNLINK is in the form:

: =DNLINK, hotword=hotword-name, (object-specification>, (DN-object-schema>, (DN-link-schema>

: = title = object-name [, object-name]*

<DN-object-schema) <link-schema)

: =DNobjectSchema = object-schema-name : =DNlinkSchema=link-schema-name

This group of statements should be understood as follows:

• Object specification identifies one or more objects, i.e. the destination nodes.
• The computations specified in the DN-object-schema are applied to information in the source node. The result a logical frame.
• The computations specified in the DN-link-schema are applied to this logical frame to produce active pipes. Thl is one active pipe per destination object specified in the DNLINK statement.
• The data in an active pipe is used to elaborate the corresponding destination node, and a part or whole of it II be passed to executable components within that destination node for processing.

6.1. Application motivations for DNLINK

The major motivation for introducing DNLINK links was to more effectively use HyperNet in Software Engineeri environments. HyperNet is already used in the software development process in number of ways, including browsi of software reuse libraries (Marovac, 1992a). However, in all current applications Hypernet is used as a passive t( in the sense that it helps with organization, maintenance and browsing of source code of a program under developme Introduction of DNLINK links allow us to develop execution environments in which a software engineer can requ on demand execution of modules in the program, even when these modules are distributed. These links also provi us with means to to develop a program testing environment in which the engineer can interactively initiate deb run of a single module or a subtree of modules and interactively link appropriate test sets for each run.

Furthermore, we are researching into the development and implementation of Module Interconnection Langual on the top of HyperNet and its Scripts language.

7. BILINK ACTIVE LINKS

These links are a logical combination of DNLINK and UPLINK links. They project active computations in be direction, and these computations affect the source nodes and the destination nodes. The effects are as described the sections on UPLINK and DNLINK links. Therefore, we will not dwell on BILINK links. They are defined a~

: = BILINK, <hotword-specification), < object-specification), <UP-object-schema), <UP-link-schema) <DN-object-schema), <DN-link-schema)

The sequence of the BILINK associated computations is divided into the two stages, i.e. the UP-schemas a DN-schemas are not performed in the same time. During the elaboration of the source node the calculations specifJ in the UP-schemas are performed as described in the UPLINK section. When the elaboration of the source no into the source frame is completed, and the user clicks on the hotword associated with the BILINK then t computations specified in the in DN-schemas are performed. The computations in DN-schemas could effect chanl in the active view in the source frame.