Abinitio Glossary

AB INITIO GLOSSARY

Ad-hoc Multifile: A parallel dataset created by naming a set of serial files as its partitions. These partitions can be named by explicitly listing the serial files, or by using a shell expression that expands at runtime to a list of serial files.

Co>Operating System: The Co>Operating System is core software that unites a network of computing resources-CPUs, storage disks, programs, datasets-into a production-quality data processing system with scalable performance and mainframe reliability. The Co>Operating System is layered on top of the native operating systems of a collection of servers. It provides a distributed model for process execution, file management, process monitoring, checkpointing, and debugging.

Component parallelism: A graph with multiple processes running simultaneously on separate data uses component parallelism.

Checkpoint is a phase that acts as an intermediate stopping point in a graph to safeguard against failures. By assigning phases with checkpoints to a graph, you can recover completed stages of the graph if failure occurs.

Control partition is the file in a multifile that contains the locations (URLs) of the multfile's data partitions.

Data parallelism: A graph that deals with data divided into segments and operates on each segment simultaneously uses data parallelism. Nearly all commercial data processing tasks can use data parallelism. To support this form of parallelism, Ab Initio provides Partition components to segment data, and Departition components to merge segmented data back together.

Deadlock occurs when a program cannot progress. It depends on the patterns of the data and typically occurs in graphs with data flows that split and then join.

A graph carries a potential deadlock when flows diverge and converge within the same phase. If the flows converge at a component that reads its input flows in a particular order, that component may wait for records to arrive on one flow even as the unread data accumulates on others because components have a limited buffering capacity.

DML is an acronym for Data Manipulation Language. It is the Ab Initio programming language you use to define record formats (which are kinds of types), expressions, transform functions, and key specifiers. Expressions include a large number of built-in functions. Files with .dml extensions contain record format definitions.

Export: You can export or pass on properties (parameters, layouts, or ports) from components to graphs or subgraphs. This creates a graph level parameter that is referenced by the original parameter in the component.

If you export properties from two different components and give them the same name, the properties will have the same value. A typical use is to export a key parameter from a Partition by Key and a Sort, so they have the same value.

Fan-in flows connect components with a large number of partitions to components with a smaller number of partitions. The most common use of fan-in is to connect flows to Departition components. This flowpattern is used to merge data divided into many segments back into a single segment, so other programs can access the data.

When you connect a component running in parallel to any component via a fan-in flow, the number of partitions of the original component must be a multiple of the number of partitions of the component receiving the fan-in flow. For example, you can connect a component running 9 ways parallel to a component running 3 ways parallel, but not to a component running 4 ways parallel.

To deal with the latter case, Repartition the data by inserting one of the Partition components between the components. This turns the fan-in-flow into an All-to-all flow, allowing a record from any partition of one component to flow into any partition of the other component.

Fan-out flows connect components with a small number of partitions to components with a larger number of partitions. The most common use of fan-out is to connect flows from partition components. This flow pattern is used to divide data into many segments for performance improvements.

When you connect a Partition component running in parallel to another component running in parallel via a fan-out flow, the number of partitions of the component receiving the fan-out flow must be a multiple of the number of partitions of the Partition component. For example, you can connect a Partition component with 3 partitions via a fan-out flow to a component with 9 partitions, but not to a component with 10 partitions.

To deal with the later case, Repartition the data by inserting a Departition component after the Partition component. This turns the fan-out flow into an All-to-all flow, allowing a record from any partition of the Partition component to flow into any partition of the target component.

Flow: A flow carries a stream of data between components in a graph. Flows connect components via ports. Ab Initio supplies four kinds of flows with different patterns: straight, fan-in, fan-out, and all-to-all.

Graph is a diagram that defines the various processing stages of a task and the streams of data as they move from one stage to another. Visually, stages are represented by components and streams are represented by flows. The collection of components and flows comprise an Ab Initio graph. You can create graphs from the main menu of the GDE.

GDE: The Graphical Development Environment (GDE) provides a graphical user interface into the services of the Co>Operating System.

Layout: A layout is a list of host and directory locations, usually given by the URL of a multifile. If the locations are not in a multifile, the layout is a list of URLs called a custom layout.

A program component's layout lists the hosts and directories in which the component runs. A dataset component's layout lists the hosts and directories in which the data resides. Layouts are set on the Properties Layout tab.

The layout defines the level of parallelism. Parallelism is achieved by partitioning data and computation across processors.

Ab Initio uses layout markers to show the level of parallelism on components. If the GDE can determine the level of parallelism, it uses that level as the marker. For example, non-parallel files have a marker of 1. If the GDE cannot determine the level of parallelism, it abbreviates layouts as L1, L2, and so on. An asterisk next to a layout marker (L1*) indicates a propagated layout.

.mdc file: A file with an .mdc file extension represents a Dataset or custom dataset component.

Multifile is a parallel file that is composed of individual files on different disks and/or nodes. The individual files are partitions of the multifile. Each multifile contains one control partition and one or more data partitions. Multifiles are stored in distributed directories called multidirectories.

The data in a multifile is usually divided across partitions by one of these methods:
Random or roundrobin partitioning
Partitioning based on ranges or functions, or
Replication or broadcast, in which each partition is an identical copy of the serial data.

A partition is a file that is a portion of a multifile. A partition is a segment of a parallel computation.

Phase is a stage of a graph that runs to completion before the start of the next stage. By dividing a graph into phases, you can save resources, avoid deadlock, and safeguard against failures.

If a graph has deadlock potential, for example in a merge-split situation, use different phases in the upstream and downstream components to stagger the reading and writing to disk.

To protect a graph, all phases are checkpoints by default. A checkpoint is a special kind of phase that saves status information and allows you to recover from failures.

Pipeline Parallelism: A graph with multiple components running simultaneously on the same data uses pipeline parallelism.

Each component in the pipeline continuously reads from upstream components, processes data, and writes to downstream components. Since a downstream component can process records previously written by an upstream component, both components can operate in parallel.

Record format is either a DML file or a DML string that describes data. You set record formats on the Properties Parameters tab of the Properties dialog box. A record format is a type applied to a port.

Repartition data is to change the degree of parallelism or the grouping of partitioned data. For instance, if you have divided skewed data, you can repartition using a Partition by Key connected to a Gather with an All-to-all flow.

Sandbox: A sandbox is a collection of graphs and related files that are stored in a single directory tree, and treated as a group for purposes of version control, navigation, and migration. A sandbox can be a file system copy of a repository project.

Skew refers to lopsided data storage or program execution. Causes of skew range from uneven input data to different loads on different processors. Repartitioning often alleviates skewed input data. Modifying the layouts often repairs skewed program execution. Skew for a data storage partition is defined as:
(N- AVERAGE) /MAX
Where: N is the number of bytes in that partition
AVERAGE is the total number of bytes in all the partitions divided by number of partitions
MAX is the number of bytes in the partition with the most bytes.

Skew for program execution is similar, using CPU seconds instead of number of bytes.

Straight flows: connect components that have the same number of partitions. Partitions of components connected with straight flows have a one-to-one correspondence. Straight flows are the most common flow pattern.

Subgraph is a graph fragment. Just like graphs, subgraphs can contain components and flows. Subgraphs are useful for grouping a graph into subtasks and reusing them.

Watcher: A watcher lets you view the data that has passed through a Flow.
To use a watcher, do the following:
Turn on debugging mode.
Add a watcher on a flow.
Run the graph.
View the data.