[go: nahoru, domu]

US7089385B1 - Tracking in-progress writes through use of multi-column bitmaps - Google Patents

Tracking in-progress writes through use of multi-column bitmaps Download PDF

Info

Publication number
US7089385B1
US7089385B1 US11/068,545 US6854505A US7089385B1 US 7089385 B1 US7089385 B1 US 7089385B1 US 6854505 A US6854505 A US 6854505A US 7089385 B1 US7089385 B1 US 7089385B1
Authority
US
United States
Prior art keywords
data
memory
map
data volume
state
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
US11/068,545
Inventor
Oleg Kiselev
Anand A. Kekre
John A. Colgrove
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Veritas Technologies LLC
Original Assignee
Veritas Operating Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Veritas Operating Corp filed Critical Veritas Operating Corp
Priority to US11/068,545 priority Critical patent/US7089385B1/en
Application granted granted Critical
Publication of US7089385B1 publication Critical patent/US7089385B1/en
Assigned to SYMANTEC OPERATING CORPORATION reassignment SYMANTEC OPERATING CORPORATION CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: VERITAS OPERATING CORPORATION
Assigned to VERITAS US IP HOLDINGS LLC reassignment VERITAS US IP HOLDINGS LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SYMANTEC CORPORATION
Assigned to BANK OF AMERICA, N.A., AS COLLATERAL AGENT reassignment BANK OF AMERICA, N.A., AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: VERITAS US IP HOLDINGS LLC
Assigned to WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT reassignment WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: VERITAS US IP HOLDINGS LLC
Assigned to VERITAS TECHNOLOGIES LLC reassignment VERITAS TECHNOLOGIES LLC MERGER AND CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: VERITAS TECHNOLOGIES LLC, VERITAS US IP HOLDINGS LLC
Assigned to WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT reassignment WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: VERITAS TECHNOLOGIES LLC
Assigned to VERITAS US IP HOLDINGS, LLC reassignment VERITAS US IP HOLDINGS, LLC TERMINATION AND RELEASE OF SECURITY IN PATENTS AT R/F 037891/0726 Assignors: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99953Recoverability
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99955Archiving or backup

Definitions

  • Businesses must be prepared to eliminate or minimize data loss and recover quickly with useable data after an unpredicted event such as a hardware or software failure in the data processing system.
  • an unexpected event occurs that causes loss of data
  • businesses often recreate the data using backup copies of their data made on magnetic storage tape.
  • Restoring data from a backup tape is typically a time-consuming process that often results in a substantial loss of business opportunity.
  • Business opportunity is lost because the data processing system cannot process new data transactions while data is being recreated from backup copies.
  • restoration from tape usually involves loss of data. For most businesses, this kind of data loss and down time is unacceptable.
  • Mirrored data volume and virtual point-in-time (PIT) backup technologies offer better solutions for full and accurate restoration of business critical data.
  • Mirrored data volume and virtual PIT technologies not only minimize or eliminate data loss, but also enable rapid recovery of a data processing system when compared to conventional bulk data transfer from sequential media.
  • FIG. 1 illustrates an exemplary data processing system 10 that employs mirrored data volume and virtual PIT backup technology.
  • Data processing system 10 includes a host node 12 coupled to data storage systems 14 – 18 .
  • Data storage systems 14 – 18 include data memories 24 – 28 , respectively.
  • data memory 24 stores a primary data volume
  • data memory 26 stores a virtual PIT backup copy of the primary data volume
  • data memory 28 stores a mirrored copy of the data volume.
  • Data processing system 10 shown in FIG. 1 and its description herein should not be considered prior art to the invention disclosed and/or claimed.
  • Host node 12 may take form in a computer system (e.g., a server computer system) that receives and processes requests to read or write data to the primary data volume stored within memory 24 . In response to receiving these requests, host node 12 generates read or write-data transactions for reading or writing data to the primary data volume within data memory 24 . It is noted that host node 12 is capable of accessing each of the memories 24 – 28 or memory internal to host node 12 for reading or writing data to the data volumes stored therein.
  • a computer system e.g., a server computer system
  • Each of the data memories 24 – 28 may include several memory devices such as arrays of magnetic or optical discs.
  • the primary data volume maybe distributed across several memory devices of memory 24 .
  • the virtual PIT copy of the primary data volume is distributed across several memory devices of data memory 26
  • the mirrored copy of the primary data volume is distributed across several memory devices of data memory 28 .
  • Each of the data memories 24 – 28 may include n max physical memory blocks into which data can be stored. More particularly, data memory 24 includes n max memory blocks allocated by host node 12 for storing data of the primary data volume, data memory 26 includes n max memory blocks allocated to store the virtual PIT backup copy of the primary data volume, and data memory 28 includes n max memory blocks allocated to store the mirrored copy of the primary data volume. Corresponding memory blocks in data memories 24 – 28 are equal in size. Thus, memory block 1 of data memory 24 is equal in size to memory block 1 of data memories 26 and 28 .
  • Host node 12 creates the virtual PIT backup copy of the primary data volume according to the methods described in U.S. patent application No. 10/143,059 entitled “Method and Apparatus for Creating a Virtual Data Copy,” and U.S. patent application Ser. No. 10/254,753 now U.S. Pat. No. 6,912,631, filed Sep. 25, 2002 and entitled “Method and Apparatus for Restoring a Corrupted Data Volume,” each of which is incorporated herein by reference in entirety. These references describe how host node 12 reads and writes data to the virtual PIT copies. Additionally, the above referenced applications describe how the virtual PIT backup copy can be converted to a real PIT backup copy using a background copying process executed by host node 12 .
  • Host node 12 creates the virtual PIT backup copy of the primary data volume when host node receives or internally generates a command to create a PIT copy of the primary data volume. Initially (i.e., before any virtual PIT copy is created in data memory 26 ) data memory 26 contains no data.
  • host node 12 creates the first virtual PIT backup copy in memory 26
  • host node 12 creates a pair of valid/modified (VM) maps, such as maps 30 and 32 shown in FIG. 2 .
  • FIG. 2 also shows a third map 34 which will be more fully described below.
  • VM maps 30 and 32 correspond to the primary data volume and the virtual PIT copy thereof, respectively, stored within data memories 24 and 26 , respectively.
  • VM maps 30 and 32 will be referred to as primary VM map 30 and PIT backup copy VM map 32 .
  • Each of the VM maps 30 and 32 include n max entries, each entry having two bits.
  • Each entry of primary VM map 30 corresponds to a respective block of data memory 24
  • each entry of the virtual PIT backup copy VM map 32 corresponds to a respective block of data memory 26 .
  • the first and second bits in each entry of VM maps 30 and 32 are designated V n and M n , respectively.
  • V n of each entry depending on its state (i.e., logical 0 or logical 1), indicates whether block n of the associated memory contains valid data.
  • V n of the primary VM map 30 indicates whether a corresponding memory block n in memory 24 stores valid data.
  • V 2 of VM map 30 indicates that block 2 of data memory 24 contains valid data of the primary data volume
  • V 2 of the VM map 30 indicates that block 2 of data memory 24 contains no valid data of the primary data volume
  • V n of the PIT backup copy VM map 32 indicates whether memory block n in memory 26 stores a valid point-in-time copy of data in block n of memory 24 .
  • V 2 of VM map 32 when set to logical 1, indicates that block 2 of memory 26 contains a valid copy of data that existed in block 2 of memory 24 at the time the PIT backup copy was first created or at the time the PIT backup copy was last refreshed.
  • V 2 of VM map 32 when set to logical 0, indicates that block 2 of memory 26 does not contain a valid copy of data.
  • the V n bit of PIT backup copy VM map 32 is used to determine when data in block n of memory 24 is to be copied to block n of memory 26 . More particularly, when host node 12 generates a write-data transaction for writing data to block n of memory 24 , host node 12 checks the status of V n in PIT backup copy VM map 32 .
  • V n is set to logical 0, then the PIT backup copy in memory 26 lacks a valid copy of data of block n in memory 24 .
  • data in block n of memory 24 must first be copied to block n of memory 26 . If, on the other hand, V n is set to logical 1, data can be written to block n of memory 24 in accordance with the write-data transaction without first copying data to block n of memory 26 .
  • M n in each entry of VM map 30 and PIT copy VM map 32 indicates whether data within a respective block n of the corresponding memory is modified (or new) since the time the PIT backup copy was first created or last refreshed. For example, when set to logical 1, M 3 of the primary VM map 30 , indicates that block 3 of data memory 24 contains data which has been modified since the time the PIT backup copy was first created or last refreshed. When set to logical 0, M 3 of the primary VM map of 30 indicates that data has not been modified or written to Block 3 of data memory 24 since the time the PIT backup copy was first created or refreshed.
  • each entry of PIT VM map 32 is set to logical 0 thus indicating that data memory 26 contains no valid or modified data.
  • VM of each entry within primary VM map 30 is initially set to logical 1.
  • M n of each entry in primary VM map 30 is initially set to logical 0.
  • data memory 28 stores a mirrored copy of the primary data volume.
  • a mirrored copy is considered a real time copy of the primary data volume.
  • host node 12 Each time host node 12 writes data to the primary data volume stored in memory 24 via a write-data transaction, host node also generates a transaction to write the same data to the mirrored copy stored within memory 28 .
  • the mirrored copy as its name implies, closely tracks the changes to the primary data volume.
  • the mirrored copy within memory 28 provides an immediate backup data source in the event that data storage system 14 fails as a result of hardware and/or software failure. If data storage system 14 suddenly becomes unusable or inaccessible, host node 12 can service client computer system read or write requests using the mirrored volume within data memory 28 .
  • Host node 12 is also subject to hardware and/or software failure. In other words, host node 12 may crash. When host node 12 recovers from a crash, host node 12 expects contents of the data memory 24 to be consistent with the contents of the data memory 28 . Unfortunately, host node 12 may crash after a write-data transaction is completed to data memory 24 , but before the mirrored copy can be updated with the same data of the write-data transaction. If this is to happen, the contents of the primary data copy stored in memory 24 will be inconsistent with the mirrored copy in memory 28 when host node 12 recovers from its crash.
  • DR map 34 includes n max entries corresponding to the n max memory blocks in data memories 24 and 28 . Each entry includes one bit which, when set to logical 0 or logical 1, indicates whether respective memory blocks in data memories 24 and 28 are the subject of an in-progress write-data transaction. For example, host node 12 sets DR 2 to logical 1 before host node 12 writes data to memory block 2 in data memories 24 and 28 . Setting or clearing a bit in DR map 34 requires a write (i.e., an I/O operation) to memory that stores DR map 34 . After the write transaction is completed to memory block 2 of data memories 24 and 28 , host node 12 switches the state of DR 2 back to 0 using another write operation.
  • a write i.e., an I/O operation
  • DR map 34 If host node 12 crashes, the contents of DR map 34 is preserved. After recovering from the crash, host node 12 copies the contents of memory blocks in data memory 24 to respective memory blocks in data memory 28 , or vice versa, for each respective DR n bit set to logical 1 in DR map 34 . After these memory blocks are copied, host node 12 is ensured that the primary data volume of data memory 24 is consistent with the mirrored copy of data memory 28 .
  • FIG. 3 is a flow chart illustrating operational aspects of modifying or writing data of the primary volume within data memory 24 in accordance with a write-data transaction.
  • host node 12 when host node 12 receives a request to write data to the primary volume from one of the computer systems, host node 12 generates a write-data transaction to write data to block n of memory 24 .
  • host node 12 accesses VM map 32 to determine whether V n is set to logical 1. If V n of the VM map 32 is set to logical 0, then host computer 12 , as shown in step 44 , copies the contents of block n of memory 24 to the corresponding block n in memory 26 . Thereafter, in step 46 host computer 12 sets V n of the VM map 32 to logical 1.
  • host computer system 12 Before host computer system 12 writes or modifies data of block n of memory 24 that stores a portion of the primary volume and block n of memory 28 that stores a portion of the mirrored copy, host computer system 12 must set M n and DR n of primary VM map 30 and DR map 34 , respectively, to logical 1. It is noted that M n and DR n are in separate tables, and thus have separate addresses within memory. Accordingly, host computer system 12 must generate separate writes or 10 functions to set M n and DR n to logical 1.
  • M n and DR n are set to logical 1 in steps 50 and 52 , host node 12 writes data to block n in memory 24 as shown in step 54 and writes data to block n of memory 28 .
  • the method can be implemented in a computer system and, in one embodiment, includes creating a data volume in a first memory, and creating a copy of the data volume in a second memory.
  • the computer system switches the state of first and second bits of a map entry in a memory device, wherein the state of the first and second bits are switched using a single write access to the memory device.
  • FIG. 1 is a block diagram of a data processing system
  • FIG. 2 is a block diagram of VM and DR maps created by the host node shown in FIG. 1 ;
  • FIG. 3 is a flow chart illustrating operational aspects of writing or modifying data in the primary data volume and the mirrored copy thereof, after creation of a virtual PIT backup copy of the primary data volume;
  • FIG. 4 is a block diagram of a data processing system employing one embodiment of the present invention.
  • FIG. 5 is a block diagram of VM/DR and VM maps created by the host node shown in FIG. 4 , and;
  • FIG. 6 is a flow chart illustrating operational aspects of writing or modifying data in the primary data volume and the mirrored copy thereof of the data processing system shown in FIG. 4 , after creation of a virtual PIT backup copy of the primary data volume.
  • FIG. 4 illustrates (in block diagram form) relative components of a data processing system 60 employing one embodiment of the present invention.
  • Data processing system 60 is similar to data processing system 10 of FIG. 1 .
  • Data processing system 60 includes a host node 62 coupled to data storage systems 64 – 68 .
  • the term “coupled devices” should not be limited to devices coupled directly together by a communication link. Two devices may be coupled together even though communication occurs through an intervening device.
  • Data storage systems 64 – 68 include memories 74 – 78 , respectively.
  • Memory 74 stores one or more primary data volumes.
  • memory 74 stores one primary data volume, it being understood that the present invention will not be limited thereto.
  • Memory 76 stores a point-in-time (PIT) backup copy of the primary data volume whether real or virtual, while memory 78 stores a mirrored copy of the primary data volume.
  • PIT point-in-time
  • the primary data volume (and the PIT backup and mirrored copies thereof) is a collection of files that store data. While it is said that files store data, in reality data is stored in physical blocks of memory 74 which are allocated to the files by host node 62 .
  • Host node 62 may take form in a computer system (e.g., a server computer system) that includes a data storage management system 70 in the form of software instructions executing on one or more data processors (not shown).
  • Data storage management system 70 may include a file system (not shown) and a system (not shown) for managing the distribution of volume data across several memory devices.
  • Volume ManagerTM provided by Veritas Corporation of Mountain View, Calif., is an exemplary system for managing the distribution of volume data across several memory devices.
  • Host node 62 generates read or write-data transactions in response to host node 62 receiving requests from client computer systems to read or write data to the primary data volume.
  • memories 74 – 78 store the primary data volume, the PIT backup copy of the primary data volume, and the mirrored copy of the primary data volume, respectively.
  • Data memories 74 – 78 may take form in one or more dynamic or static random access memories, one or more arrays of magnetic or optical data storage disks, or combinations thereof.
  • Data memories 74 – 78 should not be limited to the foregoing hardware components; rather, data memories 74 – 78 may take form in any hardware, software, or combination of hardware and software in which data may be accessed and persistently stored.
  • Data memories 74 – 78 may take form in a complex construction of several hardware components operating under the direction of software. The memories may take form in mirrored hardware.
  • the present invention may find use with many types of redundancy/reliability systems.
  • the present invention may be used with Redundant Array of Independent Disks (RAID) systems.
  • RAID Redundant Array of Independent Disks
  • the present invention should not be limited to use in connection with the host node of a data storage network.
  • the present invention may find use in a storage switch or in any of many distinct appliances that can be used with a data storage system.
  • Each of the data memories 74 – 78 includes n max physical blocks of memory into which data can be stored. It is noted that any or all of the memories 74 – 78 may have more than n max memory blocks. However, the first n max blocks in memories 74 – 78 are allocated by host node 62 for storing the primary data volume, the PIT backup copy of the primary data volume, and the mirrored copy of the primary data volume, respectively. Corresponding memory blocks in memories 74 – 78 are equal in size. Thus, memory block 1 of memory 74 is equal in size to memory blocks 1 in memories 76 and 78 . Each of the memory blocks within memory 74 may be equal in size to each other. Alternatively, the memory blocks in memory 74 may vary in size.
  • Host node 62 can access each of the blocks in memories 74 – 78 in response to generating read or write-data transactions.
  • the read or write-data transactions are generated by host node 62 in response to host node 62 receiving read or write requests from client computer systems coupled thereto.
  • the primary data volume stored in memory 74 is the “working” volume of data processing system 60 .
  • client computer systems request a read or write access to data, the access is directed to the primary data volume.
  • the PIT backup copy of the primary data volume within memory 76 acts as a safeguard against data corruption of the primary data volume due to a software or operator error.
  • the mirrored copy within memory 78 provides a alternative source to data in the event that data storage system 64 is rendered inaccessible to host node 62 due to hardware and/or software failure.
  • Host node creates a virtual PIT backup copy of the primary data volume by creating primary VM/DR map 80 and a PIT VM map 82 shown within FIG. 5 .
  • These maps may be stored within one of the memories 74 – 78 , in a memory within host node 62 or in a memory (not shown) attached to host node 62 .
  • the contents of each of the maps 80 and 82 is accessible by host node 62 using a read or write operation.
  • Maps 80 and 82 like maps 30 and 32 described above, include n max entries.
  • Each entry of primary VM/DR map 80 corresponds to a respective block of data memories 74 and 78
  • each entry of the VM map 82 corresponds to a respective block of data memory 76 .
  • VM map 82 is substantially similar to map 32
  • VM/DR map 80 is substantially similar to a combination of VM map 30 and DR map 34 .
  • Each entry in primary VM/DR map 80 consists of three bits designated V n , M n , and DR n .
  • V n in the primary VM/DR map indicates whether a corresponding block in memory 74 contains valid primary volume data. For example, when set to logical 1, V 2 of the primary VM/DR map 80 indicates that block 2 of memory 74 contains valid primary volume data, and when set to logical 0, V 2 of primary VM/DR map 80 indicates that block 2 of data memory 74 contains no valid primary volume data.
  • V 2 of VM map 82 when set to logical 1, indicates that block 2 of memory 76 contains a valid copy of data that existed in block 2 of memory 74 at the time the PIT backup copy was first created or at the time the PIT backup copy was last refreshed.
  • V 2 of VM map 82 when set to logical 0, indicates that block 2 of memory 76 does not contain a valid copy of data.
  • M n in each entry of primary VM/DR map 80 indicates whether data has been written to or modified in the corresponding block n of memory 74 since the last time the PIT backup copy of the primary data volume was created or refreshed. For example, when set to logical 1, M 3 of primary VM/DR map 80 indicates that data has been written to block 3 of memory 74 since the last time PIT backup copy was refreshed. Likewise, when set to logical 0, M 3 of the primary VM/DR map 80 indicates that data has not been written or modified in block 3 of the memory 74 since the last time a PIT backup copy was refreshed.
  • DR n in each entry indicates whether the corresponding block n of memories 74 and 78 are the subject of an in-progress write-data transaction. For example, when set to logical 1, DR 5 of VM/DR map 80 indicates that memory block 5 of memories 74 and 78 are the subject of an in-progress write-data transaction. When set to logical 0, DR 5 of primary VM/DR map 80 indicates that memory block 5 of memories 74 and 78 are not the subject of an in progress write transaction.
  • Multiple bits in entries of primary VM/DR map 80 can be set by host node 62 using a single write operation to memory that stores VM/DR map 80 .
  • multiple bits of an entry can be updated with a single write operation since the bits are stored adjacent to each other in the memory that stores VM/DR map 80 .
  • host node can write a multi-bit binary value to an entry of VM/DR map 80 using a single write operation thereby updating or changing one or more bits of the entry. For example, suppose V 3 , M 3 , and DR 3 of entry 3 of VM/DR map 80 are initially set to logical 1, logical 0, and logical 0, respectively.
  • entry 3 of VM/DR map 80 stores binary value “100” where the most significant bit (i.e., the left most bit) corresponds to V 3 and the least significant bit (i.e., the right most bit) corresponds to DR 3 .
  • host node 62 generates a write-data transaction for writing data to block 3 of memory 74 .
  • host node 62 can write the binary value “111” to entry 3 of VM/DR map 80 thereby changing the state of M 3 and DR 3 from logical 0 to logical 1 (V 3 is maintained at logical 1 even though V 3 may be overwritten) using a single write operation.
  • host node 12 In contrast host node 12 , described in the background section above, requires separate operations to update M 3 of VM map 30 and DR 3 of DR map 34 where one operation changes the state of M 3 of VM map 30 while the other operation changes the state of DR 3 of DR map 34 .
  • host node 62 in this example could write the binary value “x11” to entry 3 of VM/DR map 80 instead of the binary value “111.”
  • V 3 could be masked from the write operation so that it is not overwritten by the binary value “x.” Indeed, any one of the V n , M n , or DR n bits can be masked when a multi-bit binary value is written to VM/DR map.
  • multiple multi-bit entries in VM/DR map 80 can be modified by host node 62 via a single write operation. For example, suppose host node 62 generates a write-data transaction for writing data to block 4 of memory 74 after host node writes data to block 3 of memory 78 . Further suppose that entries 3 and 4 of VM/DR map 80 store binary values “111” and “100,” respectively, at the time host node 62 generates the write-data transaction for writing data to block 4 or memory 74 . DR 3 is set to logical 1 thus indicating that block 3 in memories 74 and 76 are the subject of a write data transactions when, in fact, they are not.
  • VM/DR map 80 can be updated, before or after data is modified in block 4 of memory 78 in accordance with the new write-data transaction, by writing binary values “110” and “111” to entries 3 and 4 , respectively, via a single write operation since entries 3 and 4 rather than two write operations where the first operation writes binary value “110” to entry 3 and the second operation writes binary value “111” to entry 4 .
  • Entries 3 and 4 can be updated with a single write operation since entries 3 and 4 of map 80 are adjacent to each other in the memory that stores VM/DR map 80 . It is noted that V 3 , V 4 , and M 3 are not updated in the sense that the value of these bits do not change after the single write operation to VM/DR map 80 .
  • DR 3 is properly updated to logical 0 thus indicating that block 3 in memories 74 and 78 are no longer subject to a write-data transaction
  • M 4 is updated to logical 1 thus indicating that data of block 4 in memory 74 has been modified.
  • Tracking in-progress writes begins after host node 62 receives a request to write data to the primary data volume.
  • Host node 62 identifies the block n of memory 74 that is the target of the received write request.
  • Host node 62 generates a write-data transaction for writing data to block n in step 90 .
  • Host node 62 in step 92 , accesses VM map 82 to determine the state of V n .
  • V n is set to logical 0, then host node 62 copies the data of block n in memory 74 to block n of memory 76 and subsequently sets V n of VM map 82 to logical 1 with a write operation to the memory that stores map 82 . If V n of VM map 82 is determined to be set to logical 1 in step 92 or after V n is set to logical 1 in step 96 , the process proceeds to step 100 were host node sets M n and DR n bits of VM/DR map 80 using a single write operation to memory that stores VM/DR map 80 . Thereafter in steps 102 and 104 , block n of memories 64 and 68 are updated with the write data of the request in step 90 .
  • steps 102 and 104 can be reversed in order or performed in parallel.
  • host node 62 accesses entry n of VM/DR map 80 and sets the DR n bit to logical 0, thus indicating that the data write transactions to block n of memories 74 and 78 have completed and block n of memory 74 stores valid and modified data.
  • step 100 shown in FIG. 6 requires a single access to a physical address within memory that stores the primary VM/DR map 80 , the amount of time needed to perform M n and DR n updating is substantially reduced.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Disclosed is a method and apparatus for tracking in-progress writes to a data volume and a copy thereof using a multi-column bit map. The method can be implemented in a computer system and, in one embodiment, includes creating a data volume in a first memory, and creating a copy of the data volume in a second memory. In response to the computer system receiving a request to write first data to the data volume, the computer system switches the state of first and second bits of a map entry in a memory device, wherein the state of the first and second bits are switched using a single write access to the memory device.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
The present patent application is a continuation of U.S. patent application Ser. No. 10/326,432, filed on Dec. 19, 2002, now U.S. Pat. No. 6,907,507 entitled “TRACKING IN-PROGRESS WRITES THROUGH USE OF MULTI-COLUMN BITMAPS” and is incorporated by reference herein in its entirety and for all purposes.
BACKGROUND OF THE INVENTION
Many businesses rely on large-scale data processing systems for storing and processing their data. Insurance companies, banks, brokerage firms, etc., rely heavily on data processing systems. Often the viability of a business depends on the reliability of access to data contained within its data processing system. As such, businesses seek reliable ways to consistently protect their data processing systems and the data contained therein from natural disasters, acts of terrorism, or computer hardware and/or software failures.
Businesses must be prepared to eliminate or minimize data loss and recover quickly with useable data after an unpredicted event such as a hardware or software failure in the data processing system. When an unexpected event occurs that causes loss of data, businesses often recreate the data using backup copies of their data made on magnetic storage tape. Restoring data from a backup tape is typically a time-consuming process that often results in a substantial loss of business opportunity. Business opportunity is lost because the data processing system cannot process new data transactions while data is being recreated from backup copies. Further, restoration from tape usually involves loss of data. For most businesses, this kind of data loss and down time is unacceptable. Mirrored data volume and virtual point-in-time (PIT) backup technologies offer better solutions for full and accurate restoration of business critical data. Mirrored data volume and virtual PIT technologies not only minimize or eliminate data loss, but also enable rapid recovery of a data processing system when compared to conventional bulk data transfer from sequential media.
FIG. 1 illustrates an exemplary data processing system 10 that employs mirrored data volume and virtual PIT backup technology. Data processing system 10 includes a host node 12 coupled to data storage systems 1418. Data storage systems 1418 include data memories 2428, respectively. As will be more fully described below, data memory 24 stores a primary data volume, data memory 26 stores a virtual PIT backup copy of the primary data volume, and data memory 28 stores a mirrored copy of the data volume. Data processing system 10 shown in FIG. 1 and its description herein should not be considered prior art to the invention disclosed and/or claimed.
Host node 12 may take form in a computer system (e.g., a server computer system) that receives and processes requests to read or write data to the primary data volume stored within memory 24. In response to receiving these requests, host node 12 generates read or write-data transactions for reading or writing data to the primary data volume within data memory 24. It is noted that host node 12 is capable of accessing each of the memories 2428 or memory internal to host node 12 for reading or writing data to the data volumes stored therein.
Each of the data memories 2428 may include several memory devices such as arrays of magnetic or optical discs. The primary data volume maybe distributed across several memory devices of memory 24. Likewise, the virtual PIT copy of the primary data volume is distributed across several memory devices of data memory 26, and the mirrored copy of the primary data volume is distributed across several memory devices of data memory 28. Each of the data memories 2428 may include nmax physical memory blocks into which data can be stored. More particularly, data memory 24 includes nmax memory blocks allocated by host node 12 for storing data of the primary data volume, data memory 26 includes nmax memory blocks allocated to store the virtual PIT backup copy of the primary data volume, and data memory 28 includes nmax memory blocks allocated to store the mirrored copy of the primary data volume. Corresponding memory blocks in data memories 2428 are equal in size. Thus, memory block 1 of data memory 24 is equal in size to memory block 1 of data memories 26 and 28.
Host node 12 creates the virtual PIT backup copy of the primary data volume according to the methods described in U.S. patent application No. 10/143,059 entitled “Method and Apparatus for Creating a Virtual Data Copy,” and U.S. patent application Ser. No. 10/254,753 now U.S. Pat. No. 6,912,631, filed Sep. 25, 2002 and entitled “Method and Apparatus for Restoring a Corrupted Data Volume,” each of which is incorporated herein by reference in entirety. These references describe how host node 12 reads and writes data to the virtual PIT copies. Additionally, the above referenced applications describe how the virtual PIT backup copy can be converted to a real PIT backup copy using a background copying process executed by host node 12.
Host node 12 creates the virtual PIT backup copy of the primary data volume when host node receives or internally generates a command to create a PIT copy of the primary data volume. Initially (i.e., before any virtual PIT copy is created in data memory 26) data memory 26 contains no data. When host node 12 creates the first virtual PIT backup copy in memory 26, host node 12 creates a pair of valid/modified (VM) maps, such as maps 30 and 32 shown in FIG. 2. FIG. 2 also shows a third map 34 which will be more fully described below. VM maps 30 and 32 correspond to the primary data volume and the virtual PIT copy thereof, respectively, stored within data memories 24 and 26, respectively. Hence, VM maps 30 and 32 will be referred to as primary VM map 30 and PIT backup copy VM map 32.
Each of the VM maps 30 and 32 include nmax entries, each entry having two bits. Each entry of primary VM map 30 corresponds to a respective block of data memory 24, while each entry of the virtual PIT backup copy VM map 32 corresponds to a respective block of data memory 26. The first and second bits in each entry of VM maps 30 and 32 are designated Vn and Mn, respectively. Vn of each entry, depending on its state (i.e., logical 0 or logical 1), indicates whether block n of the associated memory contains valid data. Vn of the primary VM map 30 indicates whether a corresponding memory block n in memory 24 stores valid data. For example, when set to logical 1, V2 of VM map 30 indicates that block 2 of data memory 24 contains valid data of the primary data volume, and when set to logical 0, V2 of the VM map 30 indicates that block 2 of data memory 24 contains no valid data of the primary data volume. Vn of the PIT backup copy VM map 32 indicates whether memory block n in memory 26 stores a valid point-in-time copy of data in block n of memory 24. For example, V2 of VM map 32, when set to logical 1, indicates that block 2 of memory 26 contains a valid copy of data that existed in block 2 of memory 24 at the time the PIT backup copy was first created or at the time the PIT backup copy was last refreshed. U.S. application Ser. No. 10/326,427, now U.S. Pat. No. 6,880,053, entitled “Instant Refresh of A Data Volume” describes one method of refreshing a PIT backup copy and is incorporated herein by reference in entirety. V2 of VM map 32, when set to logical 0, indicates that block 2 of memory 26 does not contain a valid copy of data. The Vn bit of PIT backup copy VM map 32 is used to determine when data in block n of memory 24 is to be copied to block n of memory 26. More particularly, when host node 12 generates a write-data transaction for writing data to block n of memory 24, host node 12 checks the status of Vn in PIT backup copy VM map 32. If Vn is set to logical 0, then the PIT backup copy in memory 26 lacks a valid copy of data of block n in memory 24. Before data can be written to block n of memory 24 in accordance with the write-data transaction, data in block n of memory 24 must first be copied to block n of memory 26. If, on the other hand, Vn is set to logical 1, data can be written to block n of memory 24 in accordance with the write-data transaction without first copying data to block n of memory 26.
Mn in each entry of VM map 30 and PIT copy VM map 32, depending upon its state, indicates whether data within a respective block n of the corresponding memory is modified (or new) since the time the PIT backup copy was first created or last refreshed. For example, when set to logical 1, M3 of the primary VM map 30, indicates that block 3 of data memory 24 contains data which has been modified since the time the PIT backup copy was first created or last refreshed. When set to logical 0, M3 of the primary VM map of 30 indicates that data has not been modified or written to Block 3 of data memory 24 since the time the PIT backup copy was first created or refreshed.
When VM maps 30 and 32 are first created by host node 12, each entry of PIT VM map 32 is set to logical 0 thus indicating that data memory 26 contains no valid or modified data. For purposes of explanation, it is presumed that each block of data memory 24 contains valid data of the primary volume. Accordingly, VM of each entry within primary VM map 30 is initially set to logical 1. Lastly, Mn of each entry in primary VM map 30 is initially set to logical 0.
As noted above, data memory 28 stores a mirrored copy of the primary data volume. A mirrored copy is considered a real time copy of the primary data volume. Each time host node 12 writes data to the primary data volume stored in memory 24 via a write-data transaction, host node also generates a transaction to write the same data to the mirrored copy stored within memory 28. Thus, the mirrored copy, as its name implies, closely tracks the changes to the primary data volume. The mirrored copy within memory 28 provides an immediate backup data source in the event that data storage system 14 fails as a result of hardware and/or software failure. If data storage system 14 suddenly becomes unusable or inaccessible, host node 12 can service client computer system read or write requests using the mirrored volume within data memory 28.
Host node 12 is also subject to hardware and/or software failure. In other words, host node 12 may crash. When host node 12 recovers from a crash, host node 12 expects contents of the data memory 24 to be consistent with the contents of the data memory 28. Unfortunately, host node 12 may crash after a write-data transaction is completed to data memory 24, but before the mirrored copy can be updated with the same data of the write-data transaction. If this is to happen, the contents of the primary data copy stored in memory 24 will be inconsistent with the mirrored copy in memory 28 when host node 12 recovers from its crash.
Host node 12 uses the dirty region (DR) map 34 shown in FIG. 2 to safeguard against inconsistencies between memories 24 and 28 after crash recovery. DR map 34 includes nmax entries corresponding to the nmax memory blocks in data memories 24 and 28. Each entry includes one bit which, when set to logical 0 or logical 1, indicates whether respective memory blocks in data memories 24 and 28 are the subject of an in-progress write-data transaction. For example, host node 12 sets DR2 to logical 1 before host node 12 writes data to memory block 2 in data memories 24 and 28. Setting or clearing a bit in DR map 34 requires a write (i.e., an I/O operation) to memory that stores DR map 34. After the write transaction is completed to memory block 2 of data memories 24 and 28, host node 12 switches the state of DR2 back to 0 using another write operation.
If host node 12 crashes, the contents of DR map 34 is preserved. After recovering from the crash, host node 12 copies the contents of memory blocks in data memory 24 to respective memory blocks in data memory 28, or vice versa, for each respective DRn bit set to logical 1 in DR map 34. After these memory blocks are copied, host node 12 is ensured that the primary data volume of data memory 24 is consistent with the mirrored copy of data memory 28.
Host node 12 is burdened by having to generate two separate write operations to update VM map 30 and DR map 34 with each write-data transaction to the primary data volume. More particularly, with each write-data transaction, host node 12 must generate a first write operation to update VM map 30 and a second write operation to update DR map 34. Additionally, a substantial amount of time is needed to complete separate write operations to VM map 30 and DR map 34. To illustrate, FIG. 3 is a flow chart illustrating operational aspects of modifying or writing data of the primary volume within data memory 24 in accordance with a write-data transaction. More particularly, when host node 12 receives a request to write data to the primary volume from one of the computer systems, host node 12 generates a write-data transaction to write data to block n of memory 24. In step 42, host node 12 accesses VM map 32 to determine whether Vn is set to logical 1. If Vn of the VM map 32 is set to logical 0, then host computer 12, as shown in step 44, copies the contents of block n of memory 24 to the corresponding block n in memory 26. Thereafter, in step 46 host computer 12 sets Vn of the VM map 32 to logical 1. Before host computer system 12 writes or modifies data of block n of memory 24 that stores a portion of the primary volume and block n of memory 28 that stores a portion of the mirrored copy, host computer system 12 must set Mn and DRn of primary VM map 30 and DR map 34, respectively, to logical 1. It is noted that Mn and DRn are in separate tables, and thus have separate addresses within memory. Accordingly, host computer system 12 must generate separate writes or 10 functions to set Mn and DRn to logical 1. The time it takes, for example, for the read/write head (not shown) to travel across a magnetic disc to reach the appropriate sector that stores Mn of VM map 30, followed by the time it takes for the magnetic read/write to travel across the magnetic disc to reach another sector that stores DRn of the DR map 34, may be substantial. Once Mn and DRn are set to logical 1 in steps 50 and 52, host node 12 writes data to block n in memory 24 as shown in step 54 and writes data to block n of memory 28.
SUMMARY OF THE INVENTION
Disclosed is a method and apparatus for tracking in-progress writes to a data volume and a copy thereof using a multi-column bit map. The method can be implemented in a computer system and, in one embodiment, includes creating a data volume in a first memory, and creating a copy of the data volume in a second memory. In response to the computer system receiving a request to write first data to the data volume, the computer system switches the state of first and second bits of a map entry in a memory device, wherein the state of the first and second bits are switched using a single write access to the memory device.
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings.
FIG. 1 is a block diagram of a data processing system;
FIG. 2 is a block diagram of VM and DR maps created by the host node shown in FIG. 1;
FIG. 3 is a flow chart illustrating operational aspects of writing or modifying data in the primary data volume and the mirrored copy thereof, after creation of a virtual PIT backup copy of the primary data volume;
FIG. 4 is a block diagram of a data processing system employing one embodiment of the present invention;
FIG. 5 is a block diagram of VM/DR and VM maps created by the host node shown in FIG. 4, and;
FIG. 6 is a flow chart illustrating operational aspects of writing or modifying data in the primary data volume and the mirrored copy thereof of the data processing system shown in FIG. 4, after creation of a virtual PIT backup copy of the primary data volume.
The use of the same reference symbols in different drawings indicates similar or identical items.
DETAILED DESCRIPTION
The present invention relates to an apparatus and method for tracking in-progress write I/O's through the use of multi-column bit maps. FIG. 4 illustrates (in block diagram form) relative components of a data processing system 60 employing one embodiment of the present invention. Data processing system 60 is similar to data processing system 10 of FIG. 1. Data processing system 60 includes a host node 62 coupled to data storage systems 6468. It is noted that the term “coupled devices” should not be limited to devices coupled directly together by a communication link. Two devices may be coupled together even though communication occurs through an intervening device.
Data storage systems 6468 include memories 7478, respectively. Memory 74 stores one or more primary data volumes. For purposes of explanation, memory 74 stores one primary data volume, it being understood that the present invention will not be limited thereto. Memory 76 stores a point-in-time (PIT) backup copy of the primary data volume whether real or virtual, while memory 78 stores a mirrored copy of the primary data volume.
The primary data volume (and the PIT backup and mirrored copies thereof) is a collection of files that store data. While it is said that files store data, in reality data is stored in physical blocks of memory 74 which are allocated to the files by host node 62. Host node 62 may take form in a computer system (e.g., a server computer system) that includes a data storage management system 70 in the form of software instructions executing on one or more data processors (not shown). Data storage management system 70 may include a file system (not shown) and a system (not shown) for managing the distribution of volume data across several memory devices. Volume Manager™ provided by Veritas Corporation of Mountain View, Calif., is an exemplary system for managing the distribution of volume data across several memory devices. Host node 62 generates read or write-data transactions in response to host node 62 receiving requests from client computer systems to read or write data to the primary data volume.
As noted, memories 7478 store the primary data volume, the PIT backup copy of the primary data volume, and the mirrored copy of the primary data volume, respectively. Data memories 7478 may take form in one or more dynamic or static random access memories, one or more arrays of magnetic or optical data storage disks, or combinations thereof. Data memories 7478 should not be limited to the foregoing hardware components; rather, data memories 7478 may take form in any hardware, software, or combination of hardware and software in which data may be accessed and persistently stored. Data memories 7478 may take form in a complex construction of several hardware components operating under the direction of software. The memories may take form in mirrored hardware. It is further noted that the present invention may find use with many types of redundancy/reliability systems. For example, the present invention may be used with Redundant Array of Independent Disks (RAID) systems. Moreover, the present invention should not be limited to use in connection with the host node of a data storage network. The present invention may find use in a storage switch or in any of many distinct appliances that can be used with a data storage system.
Each of the data memories 7478 includes nmax physical blocks of memory into which data can be stored. It is noted that any or all of the memories 7478 may have more than nmax memory blocks. However, the first nmax blocks in memories 7478 are allocated by host node 62 for storing the primary data volume, the PIT backup copy of the primary data volume, and the mirrored copy of the primary data volume, respectively. Corresponding memory blocks in memories 7478 are equal in size. Thus, memory block 1 of memory 74 is equal in size to memory blocks 1 in memories 76 and 78. Each of the memory blocks within memory 74 may be equal in size to each other. Alternatively, the memory blocks in memory 74 may vary in size.
Host node 62 can access each of the blocks in memories 7478 in response to generating read or write-data transactions. The read or write-data transactions are generated by host node 62 in response to host node 62 receiving read or write requests from client computer systems coupled thereto. The primary data volume stored in memory 74 is the “working” volume of data processing system 60. Thus, when client computer systems request a read or write access to data, the access is directed to the primary data volume. The PIT backup copy of the primary data volume within memory 76 acts as a safeguard against data corruption of the primary data volume due to a software or operator error. The mirrored copy within memory 78 provides a alternative source to data in the event that data storage system 64 is rendered inaccessible to host node 62 due to hardware and/or software failure.
Host node creates a virtual PIT backup copy of the primary data volume by creating primary VM/DR map 80 and a PIT VM map 82 shown within FIG. 5. These maps may be stored within one of the memories 7478, in a memory within host node 62 or in a memory (not shown) attached to host node 62. The contents of each of the maps 80 and 82 is accessible by host node 62 using a read or write operation. Maps 80 and 82, like maps 30 and 32 described above, include nmax entries. Each entry of primary VM/DR map 80 corresponds to a respective block of data memories 74 and 78, while each entry of the VM map 82 corresponds to a respective block of data memory 76. VM map 82 is substantially similar to map 32, while VM/DR map 80 is substantially similar to a combination of VM map 30 and DR map 34.
Each entry in primary VM/DR map 80 consists of three bits designated Vn, Mn, and DRn. Vn in the primary VM/DR map, depending on its state, indicates whether a corresponding block in memory 74 contains valid primary volume data. For example, when set to logical 1, V2 of the primary VM/DR map 80 indicates that block 2 of memory 74 contains valid primary volume data, and when set to logical 0, V2 of primary VM/DR map 80 indicates that block 2 of data memory 74 contains no valid primary volume data. V2 of VM map 82, when set to logical 1, indicates that block 2 of memory 76 contains a valid copy of data that existed in block 2 of memory 74 at the time the PIT backup copy was first created or at the time the PIT backup copy was last refreshed. V2 of VM map 82, when set to logical 0, indicates that block 2 of memory 76 does not contain a valid copy of data.
Mn in each entry of primary VM/DR map 80, depending upon its state, indicates whether data has been written to or modified in the corresponding block n of memory 74 since the last time the PIT backup copy of the primary data volume was created or refreshed. For example, when set to logical 1, M3 of primary VM/DR map 80 indicates that data has been written to block 3 of memory 74 since the last time PIT backup copy was refreshed. Likewise, when set to logical 0, M3 of the primary VM/DR map 80 indicates that data has not been written or modified in block 3 of the memory 74 since the last time a PIT backup copy was refreshed.
DRn in each entry, depending on its state, indicates whether the corresponding block n of memories 74 and 78 are the subject of an in-progress write-data transaction. For example, when set to logical 1, DR5 of VM/DR map 80 indicates that memory block 5 of memories 74 and 78 are the subject of an in-progress write-data transaction. When set to logical 0, DR5 of primary VM/DR map 80 indicates that memory block 5 of memories 74 and 78 are not the subject of an in progress write transaction.
Multiple bits in entries of primary VM/DR map 80 can be set by host node 62 using a single write operation to memory that stores VM/DR map 80. In one embodiment, multiple bits of an entry can be updated with a single write operation since the bits are stored adjacent to each other in the memory that stores VM/DR map 80. More particularly, host node can write a multi-bit binary value to an entry of VM/DR map 80 using a single write operation thereby updating or changing one or more bits of the entry. For example, suppose V3, M3, and DR3 of entry 3 of VM/DR map 80 are initially set to logical 1, logical 0, and logical 0, respectively. In other words, entry 3 of VM/DR map 80 stores binary value “100” where the most significant bit (i.e., the left most bit) corresponds to V3 and the least significant bit (i.e., the right most bit) corresponds to DR3. Suppose further that host node 62 generates a write-data transaction for writing data to block 3 of memory 74. Before or after host node 62 writes data to block 3, host node 62 can write the binary value “111” to entry 3 of VM/DR map 80 thereby changing the state of M3 and DR3 from logical 0 to logical 1 (V3 is maintained at logical 1 even though V3 may be overwritten) using a single write operation. In contrast host node 12, described in the background section above, requires separate operations to update M3 of VM map 30 and DR3 of DR map 34 where one operation changes the state of M3 of VM map 30 while the other operation changes the state of DR3 of DR map 34. It is noted that host node 62 in this example could write the binary value “x11” to entry 3 of VM/DR map 80 instead of the binary value “111.” In this alternative embodiment, V3 could be masked from the write operation so that it is not overwritten by the binary value “x.” Indeed, any one of the Vn, Mn, or DRn bits can be masked when a multi-bit binary value is written to VM/DR map.
In yet another alternative embodiment, multiple multi-bit entries in VM/DR map 80 can be modified by host node 62 via a single write operation. For example, suppose host node 62 generates a write-data transaction for writing data to block 4 of memory 74 after host node writes data to block 3 of memory 78. Further suppose that entries 3 and 4 of VM/DR map 80 store binary values “111” and “100,” respectively, at the time host node 62 generates the write-data transaction for writing data to block 4 or memory 74. DR3 is set to logical 1 thus indicating that block 3 in memories 74 and 76 are the subject of a write data transactions when, in fact, they are not. VM/DR map 80 can be updated, before or after data is modified in block 4 of memory 78 in accordance with the new write-data transaction, by writing binary values “110” and “111” to entries 3 and 4, respectively, via a single write operation since entries 3 and 4 rather than two write operations where the first operation writes binary value “110” to entry 3 and the second operation writes binary value “111” to entry 4. Entries 3 and 4 can be updated with a single write operation since entries 3 and 4 of map 80 are adjacent to each other in the memory that stores VM/DR map 80. It is noted that V3, V4, and M3 are not updated in the sense that the value of these bits do not change after the single write operation to VM/DR map 80. After the single write operation, DR3 is properly updated to logical 0 thus indicating that block 3 in memories 74 and 78 are no longer subject to a write-data transaction, and M4 is updated to logical 1 thus indicating that data of block 4 in memory 74 has been modified.
Operational aspects of tracking in-progress writes is best understood with reference to the flow chart of FIG. 6 which shows operational aspects of one embodiment of the present invention. Tracking in-progress writes begins after host node 62 receives a request to write data to the primary data volume. Host node 62 identifies the block n of memory 74 that is the target of the received write request. Host node 62 generates a write-data transaction for writing data to block n in step 90. Host node 62, in step 92, accesses VM map 82 to determine the state of Vn. If Vn is set to logical 0, then host node 62 copies the data of block n in memory 74 to block n of memory 76 and subsequently sets Vn of VM map 82 to logical 1 with a write operation to the memory that stores map 82. If Vn of VM map 82 is determined to be set to logical 1 in step 92 or after Vn is set to logical 1 in step 96, the process proceeds to step 100 were host node sets Mn and DRn bits of VM/DR map 80 using a single write operation to memory that stores VM/DR map 80. Thereafter in steps 102 and 104, block n of memories 64 and 68 are updated with the write data of the request in step 90. It is noted that steps 102 and 104 can be reversed in order or performed in parallel. Eventually, when block n of memories 64 and 68 have been updated with the new data, host node 62, in asynchronous fashion, accesses entry n of VM/DR map 80 and sets the DRn bit to logical 0, thus indicating that the data write transactions to block n of memories 74 and 78 have completed and block n of memory 74 stores valid and modified data.
Because only one write operation is needed to update the Mn and DRn bits of primary VM/DR map 80, the amount of processing overhead on host node 62 is reduced when compared to host node 12 in FIG. 1. Moreover, because write operation of step 100 shown in FIG. 6 requires a single access to a physical address within memory that stores the primary VM/DR map 80, the amount of time needed to perform Mn and DRn updating is substantially reduced.

Claims (11)

1. A method comprising:
creating a data volume;
creating a copy of the data volume;
a computer system receiving a request to write first data to the data volume;
switching the state of a pair of bits in a memory device in response to the computer system receiving the request to write first data to the data volume, wherein the state of the pair of bits are switched using a single write access to the memory device.
2. The method of claim 1 further comprising allocating memory of the memory device for a map, wherein the map comprises an entry, wherein the entry comprises the pair of bits.
3. The method of claim 1 further comprising:
writing the first data to a memory block in a first memory, wherein the first memory is configured to store the data volume;
writing the first data to a memory block in a second memory, wherein the second memory is configured to store the data volume copy;
switching the state of one or the pair bits back to its original state after writing the first data to the memory blocks of the first and second memory.
4. The method of claim 3:
wherein a first bit of the pair, when switched to a first or second state, indicates that data has been written to the first memory after allocation of the memory for the map;
wherein a second bit of the pair, when switched to a first or second state, indicates that the first and second memories will be subject to a write operation.
5. A computer readable medium comprising instructions executable by a computer system, wherein the computer system performs a method in response to executing the instructions, the method comprising:
creating a data volume;
creating a copy of the data volume;
switching the state of a pair of bits in a memory device in response to the computer system receiving a request to modify first data of the data volume, wherein the state of the pair of bits are switched using a single write access to the memory device.
6. The computer readable medium of claim 5, wherein the method further comprises allocating memory of a memory device for a map, wherein the map comprises an entry, wherein the entry comprises the pair of bits.
7. The computer readable medium of claim 6 wherein the method further comprises:
modifying the first data, wherein the first data is stored in a memory block in a first memory, wherein the first memory is configured to store the data volume;
modifying data in a second memory, wherein the second memory is configured to store the data volume copy;
switching the state of one bit of the pair back to its original state after modifying data in the second memory.
8. The computer readable medium of claim 6:
wherein one bit of the pair, when switched to a first or second state, indicates that data in the first memory was modified after allocation of the memory for the map;
wherein the other bit of the pair, when switched to a first or second state, indicates that the first and second memories are subject to write operation.
9. The computer readable medium of claim 6 wherein the method further comprises:
creating a map in the memory device, wherein the map comprises a plurality of entries, each entry comprising first, second, and third bits;
wherein each entry corresponds to a respective memory block of the data volume;
wherein each first bit, when set to a first or second state, indicates whether its respective memory block of the data volume stores valid data;
wherein each second bit, when set to a first or second state, indicates whether its respective memory block of the data volume stores data which has been modified since creation of the data volume copy;
wherein each third bit, when set to a first or second state, indicates whether its respective memory block of the data volume will be subject to a write operation, and;
wherein the pair of bits are the second and third bits, respectively, of one entry of the map.
10. A computer readable medium comprising instructions executable by a computer system, wherein the computer system performs a method in response to executing the instructions, the method comprising:
creating a data volume;
creating a mirror of the data volume;
creating a map in a memory device, wherein the map comprises a plurality of entries, wherein each entry comprises first and second bits;
wherein each first bit, when set to a first or second state, indicates whether data has been written to a respective block of the data volume since allocation of memory for the map;
wherein each second bit, when set to a first or second state, indicates whether respective blocks in the data volume and mirror will be subject to separate in-progress write operations.
11. The computer readable medium of claim 10 wherein each of the plurality of entries comprises a third bit, wherein the third bit, when set to a first or second state, indicates whether respective blocks in the data volume contains valid data.
US11/068,545 2002-12-19 2005-02-28 Tracking in-progress writes through use of multi-column bitmaps Expired - Lifetime US7089385B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/068,545 US7089385B1 (en) 2002-12-19 2005-02-28 Tracking in-progress writes through use of multi-column bitmaps

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/326,432 US6907507B1 (en) 2002-12-19 2002-12-19 Tracking in-progress writes through use of multi-column bitmaps
US11/068,545 US7089385B1 (en) 2002-12-19 2005-02-28 Tracking in-progress writes through use of multi-column bitmaps

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/326,432 Continuation US6907507B1 (en) 2002-12-19 2002-12-19 Tracking in-progress writes through use of multi-column bitmaps

Publications (1)

Publication Number Publication Date
US7089385B1 true US7089385B1 (en) 2006-08-08

Family

ID=34632682

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/326,432 Expired - Lifetime US6907507B1 (en) 2002-12-19 2002-12-19 Tracking in-progress writes through use of multi-column bitmaps
US11/068,545 Expired - Lifetime US7089385B1 (en) 2002-12-19 2005-02-28 Tracking in-progress writes through use of multi-column bitmaps

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/326,432 Expired - Lifetime US6907507B1 (en) 2002-12-19 2002-12-19 Tracking in-progress writes through use of multi-column bitmaps

Country Status (1)

Country Link
US (2) US6907507B1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7406487B1 (en) * 2003-08-29 2008-07-29 Symantec Operating Corporation Method and system for performing periodic replication using a log
US7650533B1 (en) * 2006-04-20 2010-01-19 Netapp, Inc. Method and system for performing a restoration in a continuous data protection system
US20110055500A1 (en) * 2009-09-02 2011-03-03 International Business Machines Corporation Data Storage Snapshot With Reduced Copy-On-Write
WO2015057240A1 (en) 2013-10-18 2015-04-23 Hitachi Data Systems Engineering UK Limited Target-driven independent data integrity and redundancy recovery in a shared-nothing distributed storage system
US9021296B1 (en) 2013-10-18 2015-04-28 Hitachi Data Systems Engineering UK Limited Independent data integrity and redundancy recovery in a storage system
US9069784B2 (en) 2013-06-19 2015-06-30 Hitachi Data Systems Engineering UK Limited Configuring a virtual machine
US10712953B2 (en) 2017-12-13 2020-07-14 International Business Machines Corporation Management of a data written via a bus interface to a storage controller during remote copy operations

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003157189A (en) * 2001-11-20 2003-05-30 Hitachi Ltd Data multiplexing management method, computer used for this method, and storage
US6938135B1 (en) 2002-10-04 2005-08-30 Veritas Operating Corporation Incremental backup of a data volume
US6910111B1 (en) * 2002-12-20 2005-06-21 Veritas Operating Corporation Volume restoration using an accumulator map
US7216253B2 (en) * 2003-03-26 2007-05-08 Pillar Data Systems, Inc. Methods and systems for management of systems metadata
US7302536B2 (en) * 2003-06-17 2007-11-27 Hitachi, Ltd. Method and apparatus for managing replication volumes
US7493458B1 (en) * 2004-09-15 2009-02-17 Emc Corporation Two-phase snap copy
US7617259B1 (en) * 2004-12-31 2009-11-10 Symantec Operating Corporation System and method for managing redundant storage consistency at a file system level
US7415488B1 (en) * 2004-12-31 2008-08-19 Symantec Operating Corporation System and method for redundant storage consistency recovery
US20060161808A1 (en) * 2005-01-18 2006-07-20 Burkey Todd R Method, apparatus and program storage device for providing intelligent copying for faster virtual disk mirroring
US8010758B1 (en) * 2005-05-20 2011-08-30 Symantec Operating Corporation System and method for performing secondary site synchronization based on a single change map
US7603581B2 (en) * 2006-03-17 2009-10-13 International Business Machines Corporation Remote copying of updates to primary and secondary storage locations subject to a copy relationship
US7702662B2 (en) * 2007-05-16 2010-04-20 International Business Machines Corporation Method and system for handling reallocated blocks in a file system
TWI451243B (en) * 2009-01-23 2014-09-01 Infortrend Technology Inc Method and apparatus for performing volume replication using unified architecture
JP6693224B2 (en) * 2016-03-30 2020-05-13 日本電気株式会社 Information processing system, storage device, information processing method, and program
US11782610B2 (en) * 2020-01-30 2023-10-10 Seagate Technology Llc Write and compare only data storage
US12105982B1 (en) * 2023-03-17 2024-10-01 Dell Products L.P. Techniques for optimized data resynchronization between replication sites

Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0332210A2 (en) 1988-03-11 1989-09-13 Hitachi, Ltd. Backup control method and system in data processing system
US5155824A (en) 1989-05-15 1992-10-13 Motorola, Inc. System for transferring selected data words between main memory and cache with multiple data words and multiple dirty bits for each address
EP0566966A2 (en) 1992-04-20 1993-10-27 International Business Machines Corporation Method and system for incremental backup copying of data
US5497483A (en) 1992-09-23 1996-03-05 International Business Machines Corporation Method and system for track transfer control during concurrent copy operations in a data processing storage subsystem
US5506580A (en) 1989-01-13 1996-04-09 Stac Electronics, Inc. Data compression apparatus and method
US5532694A (en) 1989-01-13 1996-07-02 Stac Electronics, Inc. Data compression apparatus and method using matching string searching and Huffman encoding
US5574874A (en) 1992-11-03 1996-11-12 Tolsys Limited Method for implementing a checkpoint between pairs of memory locations using two indicators to indicate the status of each associated pair of memory locations
US5649152A (en) 1994-10-13 1997-07-15 Vinca Corporation Method and system for providing a static snapshot of data stored on a mass storage system
US5778395A (en) 1995-10-23 1998-07-07 Stac, Inc. System for backing up files from disk volumes on multiple nodes of a computer network
US5835953A (en) 1994-10-13 1998-11-10 Vinca Corporation Backup system that takes a snapshot of the locations in a mass storage device that has been identified for updating prior to updating
US5907672A (en) 1995-10-04 1999-05-25 Stac, Inc. System for backing up computer disk volumes with error remapping of flawed memory addresses
US6141734A (en) 1998-02-03 2000-10-31 Compaq Computer Corporation Method and apparatus for optimizing the performance of LDxL and STxC interlock instructions in the context of a write invalidate protocol
US6189079B1 (en) 1998-05-22 2001-02-13 International Business Machines Corporation Data copy between peer-to-peer controllers
EP1091283A2 (en) 1999-09-30 2001-04-11 Fujitsu Limited Copying method between logical disks, disk-storage system and program for the same
US6269431B1 (en) 1998-08-13 2001-07-31 Emc Corporation Virtual storage and block level direct access of secondary storage for recovery of backup data
US6282610B1 (en) 1997-03-31 2001-08-28 Lsi Logic Corporation Storage controller providing store-and-forward mechanism in distributed data storage system
US6341341B1 (en) 1999-12-16 2002-01-22 Adaptec, Inc. System and method for disk control with snapshot feature including read-write snapshot half
US6353878B1 (en) 1998-08-13 2002-03-05 Emc Corporation Remote control of backup media in a secondary storage subsystem through access to a primary storage subsystem
US6434681B1 (en) 1999-12-02 2002-08-13 Emc Corporation Snapshot copy facility for a data storage system permitting continued host read/write access
US6460054B1 (en) 1999-12-16 2002-10-01 Adaptec, Inc. System and method for data storage archive bit update after snapshot backup
US20030041220A1 (en) 2001-08-23 2003-02-27 Pavel Peleska System and method for establishing consistent memory contents in redundant systems
US6564301B1 (en) 1999-07-06 2003-05-13 Arm Limited Management of caches in a data processing apparatus
US6591351B1 (en) 2000-05-25 2003-07-08 Hitachi, Ltd. Storage system making possible data synchronization confirmation at time of asynchronous remote copy
US6785789B1 (en) * 2002-05-10 2004-08-31 Veritas Operating Corporation Method and apparatus for creating a virtual data copy
US6792518B2 (en) 2002-08-06 2004-09-14 Emc Corporation Data storage system having mata bit maps for indicating whether data blocks are invalid in snapshot copies

Patent Citations (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0332210A2 (en) 1988-03-11 1989-09-13 Hitachi, Ltd. Backup control method and system in data processing system
US5506580A (en) 1989-01-13 1996-04-09 Stac Electronics, Inc. Data compression apparatus and method
US5532694A (en) 1989-01-13 1996-07-02 Stac Electronics, Inc. Data compression apparatus and method using matching string searching and Huffman encoding
US5155824A (en) 1989-05-15 1992-10-13 Motorola, Inc. System for transferring selected data words between main memory and cache with multiple data words and multiple dirty bits for each address
EP0566966A2 (en) 1992-04-20 1993-10-27 International Business Machines Corporation Method and system for incremental backup copying of data
US5497483A (en) 1992-09-23 1996-03-05 International Business Machines Corporation Method and system for track transfer control during concurrent copy operations in a data processing storage subsystem
US5574874A (en) 1992-11-03 1996-11-12 Tolsys Limited Method for implementing a checkpoint between pairs of memory locations using two indicators to indicate the status of each associated pair of memory locations
US6085298A (en) 1994-10-13 2000-07-04 Vinca Corporation Comparing mass storage devices through digests that are representative of stored data in order to minimize data transfer
US5835953A (en) 1994-10-13 1998-11-10 Vinca Corporation Backup system that takes a snapshot of the locations in a mass storage device that has been identified for updating prior to updating
US6073222A (en) 1994-10-13 2000-06-06 Vinca Corporation Using a virtual device to access data as it previously existed in a mass data storage system
US5649152A (en) 1994-10-13 1997-07-15 Vinca Corporation Method and system for providing a static snapshot of data stored on a mass storage system
US5907672A (en) 1995-10-04 1999-05-25 Stac, Inc. System for backing up computer disk volumes with error remapping of flawed memory addresses
US5778395A (en) 1995-10-23 1998-07-07 Stac, Inc. System for backing up files from disk volumes on multiple nodes of a computer network
US6282610B1 (en) 1997-03-31 2001-08-28 Lsi Logic Corporation Storage controller providing store-and-forward mechanism in distributed data storage system
US6141734A (en) 1998-02-03 2000-10-31 Compaq Computer Corporation Method and apparatus for optimizing the performance of LDxL and STxC interlock instructions in the context of a write invalidate protocol
US6189079B1 (en) 1998-05-22 2001-02-13 International Business Machines Corporation Data copy between peer-to-peer controllers
US6269431B1 (en) 1998-08-13 2001-07-31 Emc Corporation Virtual storage and block level direct access of secondary storage for recovery of backup data
US6353878B1 (en) 1998-08-13 2002-03-05 Emc Corporation Remote control of backup media in a secondary storage subsystem through access to a primary storage subsystem
US6564301B1 (en) 1999-07-06 2003-05-13 Arm Limited Management of caches in a data processing apparatus
EP1091283A2 (en) 1999-09-30 2001-04-11 Fujitsu Limited Copying method between logical disks, disk-storage system and program for the same
US6434681B1 (en) 1999-12-02 2002-08-13 Emc Corporation Snapshot copy facility for a data storage system permitting continued host read/write access
US6341341B1 (en) 1999-12-16 2002-01-22 Adaptec, Inc. System and method for disk control with snapshot feature including read-write snapshot half
US6460054B1 (en) 1999-12-16 2002-10-01 Adaptec, Inc. System and method for data storage archive bit update after snapshot backup
US6591351B1 (en) 2000-05-25 2003-07-08 Hitachi, Ltd. Storage system making possible data synchronization confirmation at time of asynchronous remote copy
US20030041220A1 (en) 2001-08-23 2003-02-27 Pavel Peleska System and method for establishing consistent memory contents in redundant systems
US6785789B1 (en) * 2002-05-10 2004-08-31 Veritas Operating Corporation Method and apparatus for creating a virtual data copy
US6792518B2 (en) 2002-08-06 2004-09-14 Emc Corporation Data storage system having mata bit maps for indicating whether data blocks are invalid in snapshot copies

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7406487B1 (en) * 2003-08-29 2008-07-29 Symantec Operating Corporation Method and system for performing periodic replication using a log
US7650533B1 (en) * 2006-04-20 2010-01-19 Netapp, Inc. Method and system for performing a restoration in a continuous data protection system
US20110055500A1 (en) * 2009-09-02 2011-03-03 International Business Machines Corporation Data Storage Snapshot With Reduced Copy-On-Write
US8645647B2 (en) 2009-09-02 2014-02-04 International Business Machines Corporation Data storage snapshot with reduced copy-on-write
US9069784B2 (en) 2013-06-19 2015-06-30 Hitachi Data Systems Engineering UK Limited Configuring a virtual machine
US9110719B2 (en) 2013-06-19 2015-08-18 Hitachi Data Systems Engineering UK Limited Decentralized distributed computing system
US9304821B2 (en) 2013-06-19 2016-04-05 Hitachi Data Systems Engineering UK Limited Locating file data from a mapping file
US9021296B1 (en) 2013-10-18 2015-04-28 Hitachi Data Systems Engineering UK Limited Independent data integrity and redundancy recovery in a storage system
WO2015057240A1 (en) 2013-10-18 2015-04-23 Hitachi Data Systems Engineering UK Limited Target-driven independent data integrity and redundancy recovery in a shared-nothing distributed storage system
EP2953025A1 (en) 2013-10-18 2015-12-09 Hitachi Data Systems Engineering UK Limited Target-driven independent data integrity and redundancy recovery in a shared-nothing distributed storage system
EP2953026A1 (en) 2013-10-18 2015-12-09 Hitachi Data Systems Engineering UK Limited Target-driven independent data integrity and redundancy recovery in a shared-nothing distributed storage system
US9235581B2 (en) 2013-10-18 2016-01-12 Hitachi Data Systems Engineering UK Limited Data configuration and migration in a cluster system
US9430484B2 (en) 2013-10-18 2016-08-30 Hitachi, Ltd. Data redundancy in a cluster system
US10712953B2 (en) 2017-12-13 2020-07-14 International Business Machines Corporation Management of a data written via a bus interface to a storage controller during remote copy operations

Also Published As

Publication number Publication date
US6907507B1 (en) 2005-06-14

Similar Documents

Publication Publication Date Title
US7089385B1 (en) Tracking in-progress writes through use of multi-column bitmaps
US6938135B1 (en) Incremental backup of a data volume
US7337288B2 (en) Instant refresh of a data volume copy
US7743227B1 (en) Volume restoration using an accumulator map
US7293146B1 (en) Method and apparatus for restoring a corrupted data volume
US7634594B1 (en) System and method for identifying block-level write operations to be transferred to a secondary site during replication
US7194487B1 (en) System and method for recording the order of a change caused by restoring a primary volume during ongoing replication of the primary volume
Stonebraker et al. Distributed Raid: A new Multiple copy Algorithm
US6067635A (en) Preservation of data integrity in a raid storage device
US6460054B1 (en) System and method for data storage archive bit update after snapshot backup
EP0566966B1 (en) Method and system for incremental backup copying of data
US7143249B2 (en) Resynchronization of mirrored storage devices
US5497483A (en) Method and system for track transfer control during concurrent copy operations in a data processing storage subsystem
US20020194529A1 (en) Resynchronization of mirrored storage devices
US7266654B2 (en) Storage system, server apparatus, and method for creating a plurality of snapshots
US8615641B2 (en) System and method for differential backup
US20120117028A1 (en) Load balancing backup jobs in a virtualized storage system having a plurality of physical nodes
US20060136691A1 (en) Method to perform parallel data migration in a clustered storage environment
US7549032B1 (en) Using data copies for redundancy
US7093090B1 (en) Method for creating a virtual data copy of a volume being restored
US5504857A (en) Highly available fault tolerant relocation of storage with atomicity
CN101436151A (en) Data real time backup method and system based on file system
US7350042B1 (en) Method of optimizing the space and improving the write performance of volumes with multiple virtual copies
US8140886B2 (en) Apparatus, system, and method for virtual storage access method volume data set recovery
US7165160B2 (en) Computing system with memory mirroring and snapshot reliability

Legal Events

Date Code Title Description
STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: SYMANTEC OPERATING CORPORATION, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:VERITAS OPERATING CORPORATION;REEL/FRAME:019899/0213

Effective date: 20061028

Owner name: SYMANTEC OPERATING CORPORATION,CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:VERITAS OPERATING CORPORATION;REEL/FRAME:019899/0213

Effective date: 20061028

FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: VERITAS US IP HOLDINGS LLC, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SYMANTEC CORPORATION;REEL/FRAME:037697/0412

Effective date: 20160129

AS Assignment

Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT, CONNECTICUT

Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0726

Effective date: 20160129

Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH CAROLINA

Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0001

Effective date: 20160129

Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH

Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0001

Effective date: 20160129

Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATE

Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS US IP HOLDINGS LLC;REEL/FRAME:037891/0726

Effective date: 20160129

AS Assignment

Owner name: VERITAS TECHNOLOGIES LLC, CALIFORNIA

Free format text: MERGER AND CHANGE OF NAME;ASSIGNORS:VERITAS US IP HOLDINGS LLC;VERITAS TECHNOLOGIES LLC;REEL/FRAME:038455/0752

Effective date: 20160329

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553)

Year of fee payment: 12

AS Assignment

Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT, DELAWARE

Free format text: SECURITY INTEREST;ASSIGNOR:VERITAS TECHNOLOGIES LLC;REEL/FRAME:054370/0134

Effective date: 20200820

AS Assignment

Owner name: VERITAS US IP HOLDINGS, LLC, CALIFORNIA

Free format text: TERMINATION AND RELEASE OF SECURITY IN PATENTS AT R/F 037891/0726;ASSIGNOR:WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT;REEL/FRAME:054535/0814

Effective date: 20201127