The Architectural Principles of Attunity Replicate: Support for Full Load & CDC Replication

The architecture of Attunity Replicate is comprised of three domains: the source database, the replication server, and the target database. End users interact with these domains through the Attunity Click- 2-Replicate designer and the Attunity Replicate Console. This product architecture reflects Attunity’s commitment to unlocking information assets through high performance, scalable, and easy-to-use solutions.

Support for Full Load and CDC Replication

With Attunity Replicate, the Full Load process is comprised of three steps:

1. Creating files or tables at the target database,
2. Automatically defining the metadata that is required at the target, and
3. Populating the tables with data from the source.

Unlike the CDC process, data is loaded into one or multiple tables or files at a time. This is done for efficiency reasons. Although the source tables may be subject to update activity during the Full Load process, there is no need to stop applications in the source. To guarantee the consistency of the data, the CDC process is automatically activated as soon as the Load starts.

However, changes are not applied at the target until after the table loading completes. Although the data on the target may not be consistent while the Load is active, the target data’s consistency and integrity at the conclusion of the Load are guaranteed. In addition, the Load process can be interrupted. When it restarts, it will continue from wherever it was stopped. New tables can be added to an existing target without reloading the existing tables. Similarly, columns in previously populated target tables can be added or dropped without the need to reload.

The CDC process obtains a stream of filtered events or changes in data or metadata from the transaction and archive log files. One of its most important functions is to buffer all the changes for a given transaction into a single unit before it is sent forward to the target when the transaction commits. As already mentioned above, during the initial load, CDC also buffers all the changes that occur within a transaction until after all affected tables have finished being loaded. If changes cannot be applied to the target database in a reasonable timeframe, they are buffered on the replication server for as long as necessary. This alleviates the need for re-reading the source database logs, which could take significant amounts of time.

Two important processes that are essential for both Full Load and CDC are Filter/Compress and Transformation:

• Filter/Compress — If filtering conditions are defined on the values of one or more source columns, rows and columns which are not relevant are discarded before replicating them to the target database. This may occur, for example, when a column is not in the target database schema or when a row does not pass the user-defined predicates on the rows within the source tables.
• Transformation — There may be circumstances where data to be included in the replicated tables is not an exact copy of the source data. Attunity Replicate allows users to define and automatically apply those changes to tables and columns. Examples include renaming the target table, renaming any column in the target table, deleting a target column, changing the data type and/or the length of any target column, and adding additional target columns. Attunity Replicate performs data type transformations as required, calculating the values of computed fields and applying the changes as one transaction to the target. When no user defined transformation is set, but replication is done between heterogeneous databases, some transformation between different data base data types may be required. In these cases, Attunity Replicate automatically takes care of the required transformations and computations during the Load or CDC execution.

In the next post, I'll detail Attunity Replicate's zero-footprint architecture, scalability, "Click-2-Replicate" user interface and web-based monitoring and control.