Invented by Jaidev Oppath Kochunni, Chong Liu, Hemant MISHRA, Commvault Systems Inc
The Commvault Systems Inc invention works as follows
A system that complies with certain aspects can improve the management of jobs and allocate resources for those jobs. A job manager can process and generate resource allocation requests for multiple jobs. The system also includes a resource allocator, which allocates computing resources to the plurality jobs based upon the resource allocation requests. Allocating resources can involve locking one or more resource object before allocating computing resources for the plurality. After the computing resources are allocated, the resource allocator will unlock the resource object that has been locked.
Background for Job Management and Resource Allocation in a Data Protection System
Global businesses recognize the commercial value and need to find cost-effective, reliable ways to secure their information while minimising their impact on productivity. Information protection is often an integral part of the daily work that is done within an organization. As part of its daily, weekly or monthly maintenance plan, a company may back up important computing systems like web servers, file servers, and databases. A company might also protect the computing systems of each employee, such as those used in an accounting, marketing, or engineering department.
Companies continue to look for innovative ways to manage data growth and protect data, given the ever-growing volume of data under their control. Companies often use migration techniques to move data to cheaper storage and data reduction techniques to reduce redundant data, prune lower priority data, and so forth. Companies increasingly see their stored data as an asset. Customers are increasingly looking for ways to not only manage and protect their data but also make use of it. Solutions that provide data analysis capabilities, information management and improved data presentation and accessibility features are increasingly in demand.
Many operations in a computing environment use computing resources (e.g. disk space). A backup operation might use some disk space on a secondary storage device in order to store a backup copy or another data protection copy of data that is stored in a primary storage. This allows for data to be restored if it becomes corrupted or lost. These operations may be put in a waiting status until the appropriate resources are allocated. If 1000 operations are initiated and 800 are backup operations that require resource allocation, then those 800 backup operations could be placed in the waiting status. A component (e.g. job manager) that can handle resource allocation in the computing environment might perform sequential resource allocations for the operations in this state. As the job manager allocates resources to each operation in the waiting status, the number may drop from 800 to 799 to 798 to 797 to 797 to 797 and so forth. This can lead to processing bottlenecks or inefficiencies in a busy computing environment. In some cases, additional operations that request resource allocation may be started and put in the waiting status to wait. Sometimes, the sequential processing and allocations of operations may not be sufficient to eliminate operations from the waiting states in a timely fashion. Some operations can spend too much time in the waiting status, while others, such as the job manger, may become bottlenecks.
Thus, it is desirable to have a better method for processing operations and performing resource allocations related to such operations.
A system based on certain aspects of this disclosure can improve the handling of operations and the allocation of resources associated with these operations. The system can be used to allocate resources for multiple operations according to the resource requirements of each operation. One example is that the process of allocating resources to the plurality operations is outsourced. This component is designed to handle all resource allocation requests generated in an information management system within a computing environment. In some embodiments, instead of a single job manager handling the entire job management process, including requests for resource allocation, the resource manager is responsible for resource allocation. Allocating resources can involve unlocking and locking various resource allocation tables, or other data structures that are associated with resource objects like tapes, disks, archives, and so forth. These resource objects cannot be modified or writable by any other components during resource allocation. Because resource allocation is delegated directly to another resource manager, all other tasks can be carried out without interruption. They do not depend on access to the resource allocation table and are not affected by its locking. Asynchronous threaded models can be implemented by the system, in which a job manager thread as well as a resource allocation thread associated to the job manager thread work independently.
According to at least one aspect of the present disclosure, multiple resource allocation requests can be processed by the resource manager as a whole. The resource manager may group together resource allocation requests that are related to multiple data protection operations, or other jobs. The resource manager locks the resource allocation tables to process the request, processes the allocation requests and then unlocks the allocation tables.
One aspect of this disclosure describes a method to manage resource allocation in a data storage network. This may involve initiating multiple data protection job, where the application data generated by client computing devices in the system is copied to one or several secondary storage devices to create secondary copies. The resource allocation requests associated to the data jobs that request allocation of computing resources within a data storage system are generated. A job manager of a storage component that executes on one or more computing devices within a networked storage system may initiate the data protection jobs. In response to resource allocation requests, the method may also include locking one or several resource objects that are associated with computing resource within the data storage system, allocating a portion of computing resources while they are locked, unlocking one or more resource object associated with the computing resource, and locking the resource objects. A storage management component may perform the locking and allocating of computing resources as well as unlocking the locked resource objects.
The method described in paragraph 1 can include any combination of these features: the job manger is not prevented from processing the jobs.
Another aspect is that the disclosure describes a system that includes storage managers within a networked storage system. One or more processors are used to implement the storage manager. The storage manager is composed of a job manager as well as a resource allocator. The job manager is responsible for initiating a number of data protection tasks. This involves the creation of application data by client computing devices within the storage system being copied to one of more secondary storage devices within the storage system. Resource allocation requests are generated associated with data protection jobs. These requests request that computing resources be allocated to the data storage systems to perform the respective data-protection jobs. The resource allocator can lock one or several resource objects that are associated with computing resources within a data storage system, allocate computing resources to each job for the plurality, while the resource objects remain locked, and then unlock any resource objects associated.
The system described in paragraph 1 can include any combination of these features: Where the job manger is not blocked from processing the respective job protection jobs; were the one-or more resource object tables contain metadata that associate with computing resources. Where the storage manager is also configured to generate a job protection job job job from the plurality, which will be used to allocate computing resources to each job. Where the resource allocator is further configured for receiving a plurality or multiple resource allocation requests that correspond to each job protection job. and processing them in bulk so that each of storage media is available to protect data.
For the purposes of summarizing disclosure, certain aspects and novel features of inventions have been described in this document. You should understand that not all of these advantages can be achieved according to any particular embodiment. The invention can be implemented or performed in a way that maximizes one advantage or group if advantages are taught herein, but not necessarily other advantages.
Systems and methods for improving the handling of jobs and resource allocations are disclosed. These systems and methods can be found in FIGS. 2-4. Components and functionality that are required for job management or resource allocation can be programmed into information management systems, such as the ones described in FIGS. 1A-1H.
Organizations simply cannot afford to lose critical data. This is because of the growing importance of protecting and leveraging data. Protecting and managing data is becoming more difficult due to runaway data growth and other modern realities. It is imperative to have user-friendly, efficient and powerful solutions for managing and protecting data.
Depending on the organization’s size, there may be many data production sources that fall under the control of thousands, hundreds or even thousands of employees. Individual employees used to be responsible for protecting and managing their data in the past. In other cases, a patchwork of software and hardware point solutions was used. These solutions were often offered by different vendors, and sometimes had little or no interoperability.
CERTAIN embodiments described herein offer systems and methods capable to address these and other shortcomings in prior approaches by implementing unified information management across the organization. FIG. FIG. 1A illustrates one such information management systems 100. It generally includes combinations hardware and software that are used to manage and protect data and metadata generated by various computing devices within information management system 100. An organization using the information management system 100 could be a company, other business entity, educational institution, household or governmental agency.
Generally, the systems described herein may be compatible and/or provide some of the functionality of one or more U.S patents or patent application publications assigned by CommVault Systems, Inc., each which is hereby incorporated into its entirety by reference herein.
The information management software 100 can contain a wide range of computing devices. As an example, the information management software 100 could include one or more client computing device 102 and secondary storage computing device 106, as we will discuss in more detail.
Computer devices may include without limitation one or more of the following: personal computers, workstations, desktop computers or other types generally fixed computing systems like mainframe computers or minicomputers. Other computing devices include portable or mobile computing devices like laptops, tablets computers, personal information assistants, mobile phones (such a smartphones), and other mobile/portable computing devices like embedded computers, set top boxes or vehicle-mounted devices. Servers can be included in computing devices, including mail servers, file server, database servers and web servers.
In certain cases, a computing device may include virtualized and/or Cloud computing resources. A third-party cloud service provider may provide one or more virtual machines to an organization. In some cases, computing devices may include one or more virtual machines running on a physical host computing device (or “host machine?”). The organization may use one or more virtual machines to run its database server and another virtual machine as a mail server. One example is that the organization might use one virtual machine to run its database server and another as a mail server. Both virtual machines are running on the same host computer.
A virtual machine is an operating system and associated resources that is hosted on a host computer or host machine. Hypervisor is typically software and is also known as a virtual monitor, virtual machine manager or?VMM? The hypervisor acts as a bridge between the virtual machine’s hardware and its host machine. ESX Server, by VMware, Inc., of Palo Alto, Calif., is an example of hypervisor used for virtualization. Other examples include Microsoft Virtual Server, Microsoft Windows Server Hyper-V, and Sun xVM, both by Oracle America Inc., Santa Clara, Calif. In some embodiments, hypervisors may be hardware or firmware.
The hypervisor gives each virtual operating system virtual resources such as a processor, virtual memory, and virtual network devices. Each virtual machine can have one or more virtual drives. The data of virtual drives is stored by the hypervisor in files on the filesystem of the physical host machine. These files are called virtual machine disk images (in the instance of Microsoft virtual servers) and virtual machine disk files (in case of VMware virtual server). VMware’s ESX server provides the Virtual Machine File System, (VMFS), for storage of virtual machine files. Virtual machines read and write data to their virtual disks in the same manner as physical machines.
