返回
类型 综合研究 预答辩日期 2018-03-12
开始(开题)日期 2015-06-29 论文结束日期 2018-01-22
地点 东南大学九龙湖校区计算机楼370会议室 论文选题来源 973、863项目     论文字数 8.6 (万字)
题目 云环境下面向异构负载大数据应用的资源管理机制研究
主题词 云计算,资源管理,数据中心,大数据应用,异构性
摘要 在大数据时代,作为提供服务能力的载体,云计算扮演着核心角色。云计算以数据中心为依托,采用虚拟化技术向数据挖掘、人工智能等大数据应用提供海量资源。如何高效管理海量资源,在提高上层大数据应用性能的同时优化底层资源的利用率是云计算管理者所关注的核心问题。云计算资源管理包括资源规划、资源分配、资源调整三个阶段:首先根据大数据应用的负载到达强度确定物理资源池的规模,然后根据应用的资源需求在指定物理资源池分配虚拟机,最后在虚拟机运行时根据应用的负载变化动态调整资源配置。围绕资源管理的三个阶段,针对上层应用的特点对资源管理进行优化,是当今学术界和产业界所关注的研究热点。 然而,随着大数据应用类型的逐渐多样化,当前云平台中所支撑的大数据应用负载呈现显著的异构特点:(1)云环境中批处理和流式处理大数据应用并存,导致负载到达模型异构;(2)计算密集型和网络密集型大数据应用并存,导致资源消耗类型异构;(3)异步执行和同步执行模式的大数据应用并存,导致资源使用模式异构。这些大数据应用异构负载的特点给云计算资源管理带来了巨大挑战,具体体现在三个方面:(1)在资源规划阶段,由于大数据应用负载到达模型异构复杂,现有方法难以准确预测并缺乏有效应对措施,导致系统能效降低;(2)在资源分配阶段,现有技术分配大批量异质虚拟资源需要较长时间,无法快速响应大数据应用的弹性资源需求;(3)在资源调整阶段,由于大数据应用在执行过程中资源使用模式存在复杂异构的动态变化,现有方法难以准确感知,无法满足大数据应用个体敏感的性能需求。因此,如何有效针对大数据应用异构负载的特征,实现细粒度、高能效的云计算资源管理,是本论文解决的重要问题。 本博士论文面向大数据应用异构负载的特点,围绕云计算资源管理的三个阶段展开研究。第一,在资源规划阶段,针对大数据应用负载到达模型的异构性,提出一种面向动态负载的数据中心资源规划策略,为数据中心配置低能耗的预留资源,合理规划系统的最优资源配比和服务率;第二,在资源分配阶段,针对大数据应用资源消耗类型的异构性,设计一种负载快速响应的虚拟资源分配机制,利用镜像副本技术并优化虚拟机放置加速虚拟资源分配的实例化过程;第三,在资源调整阶段,针对大数据应用资源使用模式的异构性,提出了应用感知的虚拟资源在线调整机制,通过分析应用资源使用模式和执行过程的关联关系,设计只利用一阶段精确信息的资源调整算法,实现高效准确的资源调整;最后,在理论研究的基础上,设计与实现面向异构负载大数据应用的资源管理系统,部署于东南大学云计算平台支撑真实的大数据应用,以验证本文的理论成果。 综上所述,本文面向云环境中的异构负载大数据应用,提出细粒度、高能效的资源管理机制,为现代云计算数据中心提供行之有效的资源管理解决方案。随着云计算技术的不断发展,本文的研究成果将应用于大型云计算数据中心,为数据挖掘、人工智能等各类新型大数据应用提供高性能低成本的支撑平台,具有重要的理论和应用价值。
英文题目 Research on application-aware resource management for heterogeneous big data workloads in cloud environment
英文主题词 Cloud Computing, Resource Management, Data Center, Big Data Application
英文摘要 In the era of Big Data, Cloud Computing plays a key role. Relying on large data centers, it provides massive of resources for big data applications, based on virtualization technologies. How to manage such large scale of resources, in order to improve the performance of big data applications while enhancing the utilization of underlying infrastructure is the key problem that the data center operator concerns. Resource management for big data applications includes three stages: first, it determines the scale of resource pool based on the request load. Then, it allocates the resources to big data applications in the form of VMs. Finally, it dynamically adjusts the resource allocation of VMs, corresponding to the load fluctuation. How to conduct application-aware resource management is the hot resource topic for academia and industry. While, with the development of big data applications, the emerging big data applications demonstrates obvious heterogeneity, which brings the challenge for recource management. (1) The mix of batch applications and streaming applications leads to the heterogeneity of load arrival models. (2) The mix of computation-intensive and communication-intensive applications leads to the heterogeneity of resource usage types. (3) The mix of asynchronous and synchronized applications leads to the heterogeneity of resource usage patterns. The challenges come from three-fold: (1) In the stage of capacity planning, the heterogeneity of load arrival models makes the load difficult to predict, resulting in low energy efficiency. (2) Due to the heterogeneity of resource usage types, current technologies, existing techniques require a long time to spawn a large number of VMs, which can not respond the requests timely. (3) Due to the heterogeneity of resource usage patterns, existing techniques can not allocate the resources efficiently and precisely. Thus, it is utmost important to develop fine-grained, high-efficient resource management mechanism to address these issues. This dissertation studies the resource right-sizing, resource allocation, resource adjustment problem, designing effective and efficient query processing techniques for heterogeneous applications. We first propose a data center stochastic right-sizing mechanism for dynamic workloads, aiming at tuning the optimal resource partitioning ratio and service rate. Secondly, we design a resource allocation mechanism which can respond the resource requests rapidly, in which we leverage the image replication to accelerate the VM launching process. Thirdly, we propose the application-aware VM migration mechanism, in which design a lazy migration algorithm to adjust the resource allocation using only one-step-ahead traffic information. Finally, we design and implement the resource management system for heterogeneous big data applications, deploying the system in the SEUCloud data center with real-world workloads, to evaluate the effectiveness of the theoretical approaches proposed in the dissertation. Overall, a series of effective resource management mechanisms for heterogeneous applications are explored in this dissertation. With the popularity of big data applications and cloud data centers, the proposed techniques can be applied in large cloud data centers supporting a wide range of advanced applications, which are valuable in theory and practice.
学术讨论
主办单位时间地点报告人报告主题
计算机科学与工程学院 2017.06.20 九龙湖校区计算机楼370 沈典 Flowtune: Flowlet Control for Datacenter Networks
计算机科学与工程学院 2013.12.24 九龙湖校区计算机楼313 余水 Distributed Denial of Service Attack and Defense in Clouds
计算机科学与工程学院 2014.11.07 九龙湖校区计算机楼313 Lixin Gao Distributed Frameworks for Iterative Computations on Massive Datasets
ACM南京分会 2017.06.05 九龙湖校区计算机楼313 张燕咏 Towards the Internet of Medical Things
ACM南京分会 2017.10.25 九龙湖校区计算机楼313 Margaret Martonosi End of Moore’s Law Challenges and Opportunities: Computer Architecture Perspectives
计算机科学与工程学院 2016.01.19 九龙湖校区计算机楼370 沈典 Coflow and related research topics
计算机科学与工程学院 2016.09.07 九龙湖校区计算机楼370 沈典 ECN in the virtualization environment
计算机科学与工程学院 2017.03.20 九龙湖校区计算机楼370 沈典 BBR congestion control
     
学术会议
会议名称时间地点本人报告本人报告题目
The 10th International Conference on e-Business Engineering (ICEBE’ 13) September 11-13,2013 英国考文垂 Doing Better Business: Trading a Little Execution Time for High Energy Saving under SLA Constraints
The 7th International Conference on Ubi-Media Computing(U’MEDIA’ 14) July12-14,2014 蒙古乌兰巴托 Energy-efficient Resource Allocation Model with QoS Assurance for Ubiquitous and Heterogeneous Environment
International Conference on High Performance Computing and Communications(HPCC) August 20-22,2014 法国巴黎 Cost effective Virtual Machine Image Replication Management for Cloud Data Centers
The 45th International Conference on Parallel Processing (ICPP’ 16) August 12-15,2016 美国费城 AppBag : Application-aware Bandwidth Allocation for Virtual Machines in Cloud Environment
     
代表作
论文名称
AppBag : Application-aware Bandwidth Allocation for Virtual Machines in Cloud Environment
Stochastic modeling of dynamic right-sizing for energy-efficiency in cloud data centers
Energy-efficient Resource Allocation Model with QoS Assurance for Ubiquitous and Heterogeneous Envir
Cost-effective Virtual Machine Image Replication Management for Cloud Data Centers
Doing Better Business: Trading a Little Execution Time for High Energy Saving under SLA Constraints
 
答辩委员会组成信息
姓名职称导师类别工作单位是否主席备注
黄宜华 正高 教授 博导 南京大学
韦志辉 正高 教授 博导 南京理工大学
宋爱波 正高 教授 博导 东南大学
刘波 副高 副教授 博导 东南大学
东方 副高 副教授 博导 东南大学
      
答辩秘书信息
姓名职称工作单位备注
熊润群 其他 讲师 东南大学