CN109582447B

CN109582447B - Computing resource allocation method, task processing method and device

Info

Publication number: CN109582447B
Application number: CN201811195528.3A
Authority: CN
Inventors: 毛恒
Original assignee: Zhongying Youchuang Information Technology Co Ltd
Current assignee: Zhongying Youchuang Information Technology Co Ltd
Priority date: 2018-10-15
Filing date: 2018-10-15
Publication date: 2020-09-29
Anticipated expiration: 2038-10-15
Also published as: CN109582447A

Abstract

The invention provides a computing resource allocation method, a task processing method and a device, wherein the allocation method comprises the following steps: in the case that the cluster has an available resource, sending a resource locking instruction to at least one host to lock the available resource with the at least one host; receiving a resource application of a task, and allocating at least part of the locked available resources to the task according to the resource application, so as to execute the task by using a port bound to the host. The resource can be maximally utilized by the scheme.

Description

Computing resource allocation method, task processing method and device

Technical Field

The invention relates to the technical field of computers, in particular to a computing resource allocation method, a task processing method and a task processing device.

Background

In the Hadoop cluster, the resource scheduling is usually managed by Yarn, but the resource management of Yarn is not flexible. Whether fair scheduling or container scheduling is adopted, a principle of FIFO (First Input First output) or a fair sharing principle is used among a plurality of tasks in scheduling, but the two allocation principles have certain defects in use scenes. The former may be that a large task cannot apply enough resources to block a subsequent string of small tasks whose remaining resources are enough for calculation, and the latter may be that when multiple tasks concurrently apply for the small tasks, the execution efficiency of each task is greatly reduced, and the resources cannot be flexibly allocated according to priority.

One solution is that, on a device with a Hadoop client installed, cluster resources are applied in advance and bound to a certain port of the device, and subsequently, all the resources applied in advance can be acquired by sending an http request to the port to execute a calculation task.

However, although this solution can more flexibly plan the resources of the cluster, and it is ensured that the task with higher importance or priority can obtain more resource allocation, because all the pre-allocated resources are always occupied, in a scenario with more concurrent tasks, a situation may occur in which part of http servers occupy the resources but no tasks, or only part of occupied resources are used to execute the tasks, and meanwhile, a situation may occur in which part of http servers have backlogs but do not apply for the resources.

Disclosure of Invention

In view of this, the present invention provides a computing resource allocation method, a task processing method and a device thereof, so as to maximize resource utilization.

In order to achieve the purpose, the invention adopts the following scheme:

in an embodiment of the present invention, a method for allocating computing resources includes:

in the case that the cluster has an available resource, sending a resource locking instruction to at least one host to lock the available resource with the at least one host;

receiving a resource application of a task, and allocating at least part of the locked available resources to the task according to the resource application, so as to execute the task by using a port bound to the host.

In an embodiment of the present invention, a task processing method includes:

receiving a task, and distributing the available resources of the cluster to the task by using the computing resource distribution method in the embodiment;

and applying for resources from the cluster according to the allocated available resources of the cluster by using the host, and executing the task by using the applied resources.

In an embodiment of the present invention, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the method according to the above-mentioned embodiment.

In an embodiment of the invention, a computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of the above-mentioned embodiment.

According to the computing resource allocation method, the task processing method, the computing resource allocation device, the task processing system, the computer equipment and the computer readable storage medium, the available resources of the cluster are locked, and at least part of the locked available resources are allocated to the task according to the task resource application, so that the resources can be flexibly allocated according to the requirement, and different tasks can be executed without influencing each other by using one port to execute the task, so that the resources can be maximally utilized by the scheme.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:

FIG. 1 is a flow chart illustrating a computing resource allocation method according to an embodiment of the invention;

FIG. 2 is a flow chart illustrating a computing resource allocation method according to another embodiment of the invention;

FIG. 3 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application, according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application, in accordance with an embodiment of the present invention;

FIG. 5 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention;

FIG. 6 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention;

FIG. 7 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention;

FIG. 8 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention;

FIG. 9 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention;

FIG. 10 is a flowchart illustrating a task processing method according to an embodiment of the invention;

FIG. 11 is a task processing architecture diagram according to an embodiment of the present invention;

FIG. 12 is an interaction diagram of task processing according to an embodiment of the invention;

FIG. 13 is a block diagram of a computing resource allocation apparatus according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of a task processing system according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.

FIG. 1 is a flowchart illustrating a computing resource allocation method according to an embodiment of the invention. As shown in FIG. 1, a computing resource allocation method of some embodiments may include:

step S110: in the case that the cluster has an available resource, sending a resource locking instruction to at least one host to lock the available resource with the at least one host;

step S120: receiving a resource application of a task, and allocating at least part of the locked available resources to the task according to the resource application, so as to execute the task by using a port bound to the host.

In the above step S110, the cluster may be a Hadoop cluster. After receiving a resource application of a task, a server instance may be started on a host, where the server instance on the host may be an http server, and at least part of the locked available resources may be allocated to the server instance according to the resource application, and a port of the host is bound to the server instance for monitoring, so as to lock the available resources by using the at least one host. The available resources may be locked in an occupied manner. The resource lock instruction may contain host information, information of the port of the host, allocation rules of the resource, and the like. The available resources may be locked using the existing ports of the host or the newly bound ports. All or part/remaining available resources of the cluster may be locked down.

Specifically, the step S110 may include: in the case that the cluster has available resources, sending a resource locking instruction to at least one host to enable each host to bind at least one port and lock the available resources together by using the at least one port bound by the host. The port is a logical partition of data distribution at the operating system level, and belongs to a transport layer and a session layer in an OSI (open system Interconnection) seven-layer model. The available resources may be locked to one port or may be scattered according to a preset allocation rule and then locked to a plurality of ports, and the plurality of ports may be ports bound to the same host or different hosts. In other words, the at least one port of the at least one host apportions locking the available resources.

More specifically, the step S110 may include: and under the condition that the cluster has available resources, determining at least one host according to the load state of the host, and sending a resource locking instruction to the at least one host so that each host binds at least one port and locks the available resources together by using the at least one port bound by the host. In practice, there may be many hosts and it may be determined from the host load status that a relatively idle host locks the available resources to load balance the hosts.

In step S120, the resource application of the task may be a spare computing task request, and may include a resource requirement, requester information, and the like. For example, a server side instance snoop port may be utilized to execute tasks using locked resources after receiving a task processing request. After all available resources are locked, computing resources may be allocated to the tasks according to their resource requirements, for example, the resource requirements of the task may be met by a part or all of the locked available resources, or the locked available resources and resources obtained by other methods may be used to meet the resource requirements of the task in the case that the locked available resources do not yet meet the resource requirements of the task. The allocation may be directly allocating at least part of the locked available resources to the task, or indirectly allocating at least part of the locked available resources to the task, for example, after releasing the locked available resources, reapplying the released resources to allocate to the task. Whether the resource allocated to the task is from one port or multiple ports (specifically, it can be from a server-side instance on one port or multiple ports), the task is finally executed by using one port.

In this embodiment, by locking the available resources of the cluster and allocating at least part of the locked available resources to the task according to the task resource application, the resources can be flexibly allocated according to the requirement, and by enabling one port to execute the task, different tasks can be executed without affecting each other, so that the resources can be maximally utilized by the scheme.

FIG. 2 is a flowchart illustrating a computing resource allocation method according to another embodiment of the invention. As shown in fig. 2, the method for allocating computing resources shown in fig. 1, before step S110, that is, in the case that the cluster has available resources, before sending a resource locking instruction to at least one host to lock the available resources by the at least one host, may further include:

step S130: and scanning the cluster regularly, and inquiring whether the cluster has available resources.

In this embodiment, the scanning time interval may be determined according to a history change condition of the available resources of the cluster. By scanning the cluster regularly and inquiring the residual resources, if the cluster has available resources, the resources can be occupied by a host (such as an http server) in time.

FIG. 3 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application, according to an embodiment of the present invention. As shown in fig. 3, in the step S120, allocating at least part of the locked available resources to the task according to the resource application may include:

step S1211: comparing and analyzing at least part of the available resources locked by the ports bound by the hosts with the resource requirements in the resource application;

step S1212: under the condition that the analysis result shows that at least part of the available resources locked corresponding to at least one port are consistent with the resource requirement, allocating the resources locked corresponding to the port consistent with the resource requirement to the task;

step S1213: under the condition that the analysis result shows that the sum of at least partial available resources locked corresponding to a plurality of ports meets the resource requirement, sending a first resource reapplication instruction to the corresponding host to stop the process corresponding to the plurality of ports, the sum of at least partial available resources locked corresponding to the ports meets the resource requirement, releasing the corresponding resources, reapplying the released resources to the cluster to one port bound to one host, and allocating the reapplied resources to the task;

step S1214: and under the condition that the analysis result shows that at least part of the locked available resources corresponding to each port are larger than the resource requirement, sending a second resource re-application instruction to one host to stop a process corresponding to one port, releasing corresponding resources, re-applying the released resources to the cluster to at least two ports bound to the host (which may include starting at least two server end instances and respectively monitoring one port), enabling the available resources corresponding to one port (which may include one server end instance of the two server end instances) of the at least two ports to be consistent with the resource requirement, and allocating the re-applied resources to the task.

In step S1211, a resource locked by each port (which may include each server instance started) bound by each host may be compared with a resource requirement in the resource application, and whether there is a port (which may relate to a server instance) that can meet the resource requirement of the task is analyzed. Or the sum of the resources locked by the ports can be compared with the resource requirement in the resource application, and whether a plurality of ports can meet the resource requirement of the task together is analyzed.

In step S1212, when there is exactly one port and the resource locked by the port is consistent with the resource requirement of the task, the task may be issued to the port. Where "consistent" may mean identical or close, for example, when the difference between the resource locked by the port and the resource requirement of the task is smaller than a set value, the port is considered consistent, and the set value may be configured according to actual situations.

In step S1213, the processes corresponding to the plurality of ports may be referred to as processes generated for occupying/locking available resources. When any port alone cannot meet the resource requirement of the task, the resource requirement of the task can be met by utilizing the resources locked by the ports together, wherein the ports can be multiple ports bound by the same host or multiple ports bound by different hosts. The size of the resource locked by the port can be adjusted by reapplying the resource locked by the plurality of ports, so that the task can be executed by one port.

In the step S1214, when the resource locked by each port is greater than the resource requirement of the task, the resource is reapplied based on the resource locked by one of the ports, and the reapplied resource is allocated to two or more ports, where the resource corresponding to one of the ports may be consistent with the resource requirement of the task, and the remaining resources may be reserved to one or more other ports for allocation to other tasks, so that the resource corresponding to the port may be consistent with the resource requirement of the task, and waste of the excess resources on the ports where the available resources are locked may be avoided.

In this embodiment, resources of the cluster may be dynamically allocated to the task according to resource requirements of the task according to different requirement scenarios, so that maximum utilization of the resources is achieved.

FIG. 4 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application, according to an embodiment of the present invention. As shown in fig. 4, in the step S120, receiving a resource application of a task, and allocating at least part of the locked available resources to the task according to the resource application may include:

step S1221: receiving a resource application of a task, and judging whether the locked available resource is smaller than a first set threshold value;

step S1222: under the condition that the locked available resources are smaller than the first set threshold value, judging whether the priority of the task is higher than a set level;

step S1223: under the condition that the priority of the task is higher than the set level, judging whether the resource requirement in the resource application is smaller than or equal to the locked available resource or not;

step S1224: and under the condition that the resource demand is less than or equal to the locked available resource, allocating at least part of the locked available resource to the task according to the resource application.

In step S1221, the first set threshold may be configured according to the historical task performance, for example, the remaining free resources are 20%. When the available resource that is locked is less than the first set threshold, the current cluster resource may be considered heavily loaded.

In the above step S1222, the setting level may be configured according to a history. The priority may be identified according to task type, for example, different levels may be pre-divided for different types of tasks, and when a certain type of task arrives, the corresponding level may be found according to the type of the task. Different levels may be pre-divided for different types of tasks. When the cluster resource load is heavy, the new task is not directly executed, but is firstly placed in a waiting queue and executed item by item according to the sequence of the queue. When a new task is issued each time, the priority of the task is compared with the priority of the task in the task queue in the waiting state, and the arrangement sequence of the tasks to be executed in the waiting queue is readjusted.

In this embodiment, it can be ensured that, under the condition that the cluster resource load is heavy, the tasks with higher priority and enough locked available resources are processed preferentially.

FIG. 5 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in another embodiment of the present invention. As shown in fig. 5, the method for allocating at least part of the locked available resources to the task according to the resource application shown in fig. 4 may further include:

step S1225: in the event that the priority of the task is less than or equal to the set level while the task has a delay execution tolerance, blocking the task and holding for a first set length of time or holding until at least one of the hosts releases new resources other than the available resources.

In the step S1225, there may be several methods for determining the delay tolerant duration of the task, specifically, for example: (1) presetting tolerance time length comparison tables corresponding to different task types, and obtaining time length by checking the comparison tables after receiving a task request; or, (2) different kinds of tasks are pre-configured, and different durations are respectively corresponding to different time intervals or different ranges of the load of the cluster; or, (3) at the time of task submission, the tolerance duration is explicitly specified by the task submitter. If the tolerated time length is longer than a certain set time length, the task can be considered to have a delay execution tolerance and can be blocked.

In this embodiment, when the priority of the task is low, or when the task has a certain time tolerance for delayed execution, the task may be blocked for a period of time, for example, the task is temporarily stored in or outside the host and is not processed, a new resource is released, and when the resource is empty, the task is reallocated to the port of the host for processing. In this way, more remaining resources may be reserved for higher priority tasks.

FIG. 6 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention. As shown in fig. 6, the method for allocating at least part of the locked available resources to the task according to the resource application shown in fig. 5 may further include:

step S1226: and predicting the first set time length according to the historical task execution condition of the at least one host.

In this embodiment, the historical task execution state may refer to that tasks currently being executed in each Server instance are counted, and according to the time spent by the tasks in each past periodic execution, the time spent by the current task being executed is subtracted, and after about how long time is estimated, the tasks may be executed, and the Server instance may be idle. By counting the historical task execution, it can be estimated how much work time remains that new resources may be released, thereby reasonably blocking the task.

FIG. 7 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in accordance with another embodiment of the present invention. As shown in fig. 7, the method for allocating at least part of the locked available resources to the task according to the resource application shown in fig. 5 may further include:

step S1227: during the task blocking period, judging whether the current total available resources of the at least one host are larger than or equal to the first set threshold value at intervals of a second set time length;

step S1228: and under the condition that the current total available resources are greater than or equal to the first set threshold value, allocating at least part of the current total available resources to the task according to the resource application.

In step S1227, the second setting may be configured as needed. The current total available resources may include locked available resources and released new resources.

In the above step S1228, according to the specific implementation of allocating at least part of the currently total available resources to the task according to the resource application, the method shown in fig. 3 may be implemented, except that the available resources also include released new resources.

In this embodiment, after the task is blocked, the task is prevented from being blocked when enough resources are available by re-determining the available resource condition every second set time interval, for example, several minutes, within the delay tolerance time.

FIG. 8 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in another embodiment of the present invention. As shown in fig. 8, the method for allocating at least part of the locked available resources to the task according to the resource application shown in fig. 4 may further include:

step S1229: and under the condition that the resource demand is greater than the locked available resources, sending a third instruction for newly applying for resources to the host with the maximum set number of idle resources, so that the host with the maximum set number of idle resources releases all the resources thereof, newly applying for the released all the resources to one port of one host, and allocating the newly applied resources to the task.

In the above step S1229, in the case that the resource requirement is greater than the locked available resource, the locked available resource cannot meet the resource requirement of the task. At this point, all resources of some of the hosts that are more idle may be freed up. The all resources may include the locked resource and the resource that is performing the task. This results in more available resources. Tasks that the host originally performed may reallocate resources or block execution. All released resources can be allocated to the same port (server side instance) through reapplication.

In this embodiment, it is ensured that the large task is executed by releasing all resources of the idle host. By allocating the re-applied resources based on the released resources to the same port, task execution can be facilitated.

FIG. 9 is a flowchart illustrating a method for allocating at least a portion of a locked available resource to a task according to a resource application in another embodiment of the present invention. As shown in fig. 9, the method for allocating at least part of the locked available resources to the task according to the resource application shown in fig. 4 may further include:

step S12210: judging whether the locked available resources are greater than or equal to a second set threshold value, wherein the second set threshold value is greater than or equal to the first set threshold value;

step S12211: under the condition that the locked available resources are greater than or equal to the second set threshold, judging whether the priority of the task is higher than the set level;

step S12212: and if the priority of the task is higher than the set level, allocating at least a part of the available resources which are more than the resource demand in the resource application by a set resource amount and locked to the task.

After step S120, the method for allocating computing resources may further include:

step S140: after the task is executed, the completion time of the task is recorded to judge whether the resources allocated to the task are solidified into default resources of the same task of the task.

In the above step S12210, the second setting threshold may be configured as needed. For example, the first set threshold is 20% of the remaining resources, the second set threshold is 40% of the remaining resources, or both the first set threshold and the second set threshold are 30% of the remaining resources. Computing resources may be considered to be more abundant when the available resources that are locked are greater than or equal to the second set threshold.

In step S12212, the set resource amount may be configured as needed. When computational resources are abundant, the resource allocation may be increased slightly automatically for higher priority tasks. In the step S140, the completion time of the task may be recorded at the same time, and if the linear influence of the resource allocation on the performance of the task is determined to be high, the allocation result is solidified into the default resource of the task.

In this embodiment, when the resources are abundant, more resources are allocated to the task with the higher priority, so that the processing efficiency of the task can be improved.

Based on the computing resource allocation method described in the foregoing embodiment, an embodiment of the present invention further provides a task processing method, and the specific implementation of the computing resource allocation method may be implemented with reference to the contents of the foregoing embodiment, and repeated details are not described again.

FIG. 10 is a flowchart illustrating a task processing method according to an embodiment of the invention. As shown in fig. 10, in some embodiments, the task processing method may include:

step S210: receiving a task, and distributing the available resources of the cluster to the task by using the computing resource distribution method in the embodiment;

step S220: and applying for resources from the cluster according to the allocated available resources of the cluster by using the host, and executing the task by using the applied resources.

In step S210, the available resources of the cluster are allocated to the task, specifically, the available resources of the cluster may be allocated to a server instance started on a host, so as to correspondingly allocate the available resources to the task. In step S220, specifically, the server instance may be started, and the instance is responsible for executing the computing task.

In the embodiment, after the task is received, the task is firstly distributed to the host to apply for the resource to execute the task, rather than the task is directly sent to the cluster to apply for the resource to be processed, the resource can be utilized to the maximum, and the task processing efficiency is higher.

In order that those skilled in the art will better understand the present invention, the following description will illustrate the practice of the invention in a specific embodiment.

FIG. 11 is a task processing architecture according to an embodiment of the invention. Referring to fig. 11, a set of computing resource allocation device is added on the basis of the http server. The computing resource allocation device is deployed in an HA (High available) manner, and the computing resource allocation device may not provide service processing and resource application, but only manage each http server (server instance).

The computing resource allocation device scans the cluster at regular time, queries the residual resources, finds out a relatively idle http server to occupy the resources after load if the cluster has available resources, and can scatter the resources according to preset rules during application and allocate the resources to different hosts or ports, wherein the rules can include grading, for example, 1Core1Gmemor applies for a plurality of resources, 5Core5Gmemory applies for a plurality of resources, and a user-defined splitting rule can be configured according to needs.

The computing resource allocation device receives the spark computing task request, and when a new request comes, finds the most appropriate resource combination according to the attached resource application: if the resource application is just satisfied by a port, the task is sent to the port; if the task needs to combine the resources of a plurality of ports, informing a corresponding http server (server end instance) to kill the process corresponding to the port, releasing the resources, then restarting one port, and applying for the resources from the cluster to execute the task; if the resources needed by the task are smaller than the resource allocation of the rest idle ports, one idle port is killed, the released resources are split into two parts, one part of the newly started ports execute the task, and the rest of the resources are reallocated to the idle ports.

The computing resource allocation device supports recording of historical task execution conditions and can estimate the remaining working time of each working port according to the historical execution conditions. And meanwhile, when a new task is applied, tolerance configuration for task execution delay time is added, if the current cluster resource is heavy in load (for example, the residual resource is less than 20%, the threshold value can be configured), the priority of the new task is low, or the tolerance for delay execution has a certain time, and new resource release is estimated in the delay time, the task is blocked for a period of time, and the task is executed after the resource release.

The computing resource allocation device supports automatic fine-tuning resource allocation, automatically slightly increases resource allocation for a task with a higher priority at the moment when resources are abundant (for example, the residual resources exceed 40% and a threshold value can be allocated), records task completion time, and determines that if the linear influence degree of the resource allocation on the performance of the task is higher, the allocation result is solidified into the default resource of the task.

The task processing method can comprise the following steps:

1) when a user submits a task application resource, the user does not directly access a Hadoop cluster, but passes through a batch of Http Server equipment (Server end examples) and an Http Server management module (Server end example management module) of HA (computing resource allocation device);

2) the computing resource allocation device arranges each Server end instance to apply for resources respectively, locks the resources on each Server end instance and binds to an access port respectively;

3) when a new Spark task is submitted to the computing resource allocation device, the computing resource allocation device finds a proper Server and a proper port, and issues the task.

FIG. 12 is an interaction diagram of task processing according to an embodiment of the invention. Referring to fig. 12, the task processing method may include:

step 1: the computing resource allocation device inquires the residual resource condition of the cluster at regular time;

step 2: the computing resource allocation device performs load balancing on the Http Server, arranges a plurality of hosts with lower loads to bind a plurality of ports, and applies for vacant resources from the cluster;

step 3: the Http mainly pulls up Http service on a specified port, applies for the cluster and locks resources;

step 4: a user submits or a timing task is started, a new Spark computing task is submitted, and a resource application is sent to the computing resource allocation device;

step 5: the computing resource allocation device checks the Http Server in the standby state and the locked resource list thereof, and starts analysis:

A) if the cluster has more vacant resources (the resource utilization rate threshold value can be matched), and the new task needs the resources;

a) the method is exactly consistent with a certain idle Http Server resource, can be directly executed, and changes to Step 6.1;

b) the sum of the resources of a plurality of idle Http servers is needed, the Http servers are informed to kill the processes, enough resources are reapplied to one port after the resources are released, and Step6.2 is carried out;

c) informing one Http Server of the resources smaller than any idle Http Server to kill the process, releasing the resources, splitting the resources into two parts for reapplication, wherein one part is consistent with the task requirement, and turning to Step 6.2;

B) if the cluster has less spare resources, the following conditions are divided:

a) the priority is lower, the task is blocked, the task is redistributed when the resources are vacant, and Step 6.3 is carried out;

b) b, the priority is higher, and the resource needed is less than or equal to the sum of the remaining idle resources at present, transfer to step A;

c) the priority is higher, and the required resource is greater than the sum of the remaining idle resources at present, reserve the least N groups of Http servers (N can be configured) of idle resources, reapply to a port after releasing the resources of other Http servers, change to Step6.2;

step 6.1: notifying the corresponding Http Server to prepare for work;

step 6.2: informing the corresponding Http Server, and reapplying after releasing the resources;

step 6.3: blocking the task and re-judging once per minute within a delay tolerant time (the threshold is included in the task application);

step 7: and (4) sending the calculation task to the specified port of the specified http server after Step6 processing, and starting to execute the task.

The task processing method of the embodiment adds a set of management framework on the basis of the http server resource management scheme of the hadoop cluster, realizes the maximization of cluster resource utilization, can release manual intervention, automatically adjusts the resource allocation rule according to different scenes, and has the following advantages: the resource application does not depend on the default rule of Yarn, and can be dynamically adjusted according to different demand scenes; on the basis of ensuring that tasks are allocated according to priorities, the cluster computing resources can be maximally utilized; the Yarn, the HttpServer and the management module have high availability, and the resource allocation of a new task cannot be influenced when any point is abnormal.

Specifically, for example, the management modules may be deployed in a distributed manner, and determine, through mutual heartbeat and communication, that a manager currently in an active state performs resource scheduling and task allocation, and when an abnormality occurs in the management module in the active state, the standby management module elects a new management module and activates the management state. The working state and the resource locking condition of the Server end instance are monitored by the management module in real time, and when a certain Server end instance is abnormal, the instance can be stopped or started again or tasks can be distributed by the management module. Besides the HA deployment of the Yarn, since the resources are applied by the Server instance in advance, even if the Yarn fails and cannot work, the Yarn only affects the new resource allocation. When the management module monitors that the state of the Yarn is abnormal, the resources which are applied by the current Server end instances can be locked, and before the state of the Yarn is recovered to be normal, the cluster resources do not need to be released and applied. Therefore, the Yarn, the Http Server and the management module can have high availability.

Based on the same inventive concept as the computing resource allocation method shown in fig. 1, the embodiment of the present invention further provides a computing resource allocation apparatus and a task processing system, as described in the following embodiments. Because the principle of the computing resource allocation device and the task processing system for solving the problems is similar to that of the computing resource allocation method, the implementation of the computing resource allocation device and the task processing system can refer to the implementation of the computing resource allocation method, and repeated parts are not described again.

FIG. 13 is a block diagram of a computing resource allocation apparatus according to an embodiment of the invention. As shown in fig. 13, the computing resource allocation apparatus of some embodiments may include: a resource locking unit 310 and a resource allocation unit 320, which are connected to each other.

A resource locking unit 310, configured to send a resource locking instruction to at least one host to lock an available resource with the at least one host if the cluster has the available resource;

a resource allocation unit 320, configured to receive a resource application of a task, and allocate at least part of the locked available resources to the task according to the resource application, so as to execute the task by using a port bound to one host.

Fig. 14 is a schematic structural diagram of a task processing system according to an embodiment of the present invention. As shown in FIG. 14, the task processing system of some embodiments may include:

the computing resource allocation device 410 according to the above embodiment is configured to receive the task and allocate the available resources of the cluster to the task;

at least one host 420, configured to apply for resources from the cluster according to the allocated available resources of the cluster, and execute the task using the applied resources.

The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the steps of the method described in the above embodiment are implemented.

Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

In summary, according to the computing resource allocation method, the task processing method, the computing resource allocation apparatus, the task processing system, the computer device, and the computer readable storage medium of the embodiments of the present invention, by locking the available resources of the cluster and allocating at least part of the locked available resources to the task according to the task resource application, the resources can be flexibly allocated according to the requirement, and by executing the task using one port, different tasks can be executed without affecting each other, so that the resources can be utilized to the maximum extent by the present solution.

In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the various embodiments is provided to schematically illustrate the practice of the invention, and the sequence of steps is not limited and can be suitably adjusted as desired.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for allocating computing resources, comprising:

receiving a resource application of a task, and allocating at least part of the locked available resources to the task according to the resource application so as to execute the task by using a port bound by the host;

the receiving a resource application of a task and allocating at least part of the locked available resources to the task according to the resource application includes:

receiving a resource application of a task, and judging whether the locked available resource is smaller than a first set threshold value;

under the condition that the locked available resources are smaller than the first set threshold value, judging whether the priority of the task is higher than a set level;

under the condition that the priority of the task is higher than the set level, judging whether the resource requirement in the resource application is smaller than or equal to the locked available resource or not;

and under the condition that the resource demand is greater than the locked available resources, sending a third instruction for newly applying for resources to the host with the maximum set number of idle resources, so that the host with the maximum set number of idle resources releases all the resources thereof, newly applying for the released all the resources to one port of one host, and allocating the newly applied resources to the task.

2. The method of computing resource allocation according to claim 1, wherein in the event that the cluster has available resources, sending a resource lock instruction to at least one host to lock the available resources with the at least one host comprises:

and under the condition that the cluster has available resources, determining at least one host according to the load state of the host, and sending a resource locking instruction to the at least one host so that each host binds at least one port and locks the available resources together by using the at least one port bound by the host.

3. The method of computing resource allocation according to claim 2, wherein allocating at least a portion of the locked available resources to the task in accordance with the resource application comprises:

comparing and analyzing at least part of the available resources locked by the ports bound by the hosts with the resource requirements in the resource application;

under the condition that the analysis result shows that at least part of the available resources locked corresponding to at least one port are consistent with the resource requirement, allocating the resources locked corresponding to the port consistent with the resource requirement to the task;

under the condition that the analysis result shows that the sum of at least partial available resources locked corresponding to a plurality of ports meets the resource requirement, sending a first resource reapplication instruction to the corresponding host to stop the process corresponding to the plurality of ports, the sum of at least partial available resources locked corresponding to the ports meets the resource requirement, releasing the corresponding resources, reapplying the released resources to the cluster to one port bound to one host, and allocating the reapplied resources to the task;

and under the condition that the analysis result shows that at least part of the locked available resources corresponding to each port are larger than the resource requirement, sending a second resource re-application instruction to one host to stop the process corresponding to one port, releasing the corresponding resources, re-applying the released resources to the cluster to at least two ports bound to the host, enabling the available resources corresponding to one port of the at least two ports to be consistent with the resource requirement, and allocating the re-applied resources to the task.

4. The method of computing resource allocation according to claim 1, wherein receiving a resource request for a task and allocating at least a portion of the locked available resources to the task based on the resource request comprises:

and under the condition that the resource demand is less than or equal to the locked available resource, allocating at least part of the locked available resource to the task according to the resource application.

5. The method of computing resource allocation according to claim 1, wherein receiving a resource request for a task and allocating at least a portion of the locked available resources to the task based on the resource request, further comprises:

in the event that the priority of the task is less than or equal to the set level while the task has a delay execution tolerance, blocking the task and holding for a first set length of time or holding until at least one of the hosts releases new resources other than the available resources.

6. The method of computing resource allocation according to claim 5, wherein receiving a resource request for a task and allocating at least a portion of the locked available resources to the task based on the resource request, further comprises:

during the task blocking period, judging whether the current total available resources of the at least one host are larger than or equal to the first set threshold value at intervals of a second set time length;

and under the condition that the current total available resources are greater than or equal to the first set threshold value, allocating at least part of the current total available resources to the task according to the resource application.

7. The computing resource allocation method of claim 1,

receiving a resource application of a task, and allocating at least part of the locked available resources to the task according to the resource application, further comprising:

judging whether the locked available resources are greater than or equal to a second set threshold value, wherein the second set threshold value is greater than or equal to the first set threshold value;

under the condition that the locked available resources are greater than or equal to the second set threshold, judging whether the priority of the task is higher than the set level;

if the priority of the task is higher than the set level, allocating at least a part of the available resources which are more than the resource demand in the resource application by a set resource amount and locked to the task;

the method further comprises the following steps:

and after the task is executed, recording the completion time of the task, and judging whether the resources distributed to the task are solidified into default resources of the same task of the task.

8. A task processing method, comprising:

receiving a task and allocating the available resources of the cluster to the task using the computing resource allocation method of any one of claims 1 to 7;

9. A computing resource allocation apparatus, comprising:

a resource locking unit, configured to send a resource locking instruction to at least one host to lock an available resource with the at least one host if a cluster has the available resource;

a resource allocation unit, configured to receive a resource application of a task, and allocate at least part of the locked available resources to the task according to the resource application, so as to execute the task using a port bound to one host;

the resource allocation unit includes:

the receiving sub-unit is used for receiving a resource application of a task and judging whether the locked available resource is smaller than a first set threshold value or not;

a priority judging and classifying unit, configured to judge whether the priority of the task is higher than a set level when the locked available resource is smaller than the first set threshold;

the resource judgment sub-unit is used for judging whether the resource requirement in the resource application is less than or equal to the locked available resource or not under the condition that the priority of the task is higher than the set level;

and the reallocation subunit is configured to, when the resource demand is greater than the locked available resource, send a third instruction for re-applying for resources to the host with the maximum set number of idle resources, so that the host with the maximum set number of idle resources releases all resources thereof, re-apply for the released all resources to one port of one host from the cluster, and allocate the re-applied resources to the task.

10. A task processing system, comprising:

the computing resource allocation apparatus of claim 9, configured to receive a task and allocate available resources of a cluster to the task;

and the at least one host is used for applying for resources from the cluster according to the distributed available resources of the cluster and executing the tasks by using the applied resources.

11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 8 are implemented when the program is executed by the processor.

12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.