Due to the proliferation of smart mobile equipment and innovative mobile data services, the wireless traffic demands are increasing at an unprecedented pace. Although multi-layer heterogeneous networks partially meet the data requirements of mobile terminals with a speed of rapid increase, the use of wireless resources (i.e., spectrum) will become even more stretched. The limited wireless resources cannot possibly meet the continuous increased requirements of various new mobile services and application cases. In addition, the reuse of wireless resources can lead to serious interference and the unreasonable resource allocations weaken the network performance. Therefore, it is urgently required to manage and allocate the wireless resources efficiently to meet the various application requirements, guarantee the quality of service (QoS), maximize the utilization of wireless resources and alleviate the shortage of resources. Since the integration of multiple networks and the emergence of new technologies make the interferences more complex in the fifth generation mobile communication (5G) network, the research on resource allocation schemes with multiple types of resources will be more urgent. This thesis focuses on resource allocation schemes in multiple layer heterogeneous wireless communication networks. For multiple layer heterogeneous networks with small cell and device-to-device (D2D) nodes, considering sum-rate, energy efficiency (EE), outage performance, user association and backhaul saving, the research of the thesis includes game theoretic max-logit learning approaches for joint base station selection and resource allocation in heterogeneous networks, resource allocation for outage performance in heterogeneous networks, resource optimization for D2D and small cell uplink communications underlaying cellular networks, energy-efficient resource allocation for energy harvesting-based D2D communication, resource optimization in heterogeneous cloud radio access networks, joint service improvement and content placement for cache-enabled heterogeneous cellular networks (HCNs) and the efficient deployment of unmanned aerial vehicle (UAV) for UAV-assisted communication networks. The main contributions of this thesis are listed as follows: (1) We investigate the problem of joint base station selection and resource allocation in an orthogonal frequency division multiple access (OFDMA) HCN. The original throughput maximization problem is NP-hard and we propose to solve it by using game theoretic stochastic learning approaches. To this end, we first transform the original problem into a tractable form which has a weighted utility function. Then we prove that an exact potential game applies and it exists the best Nash equilibria which is a near optimal solution of the original problem. In order to obtain the optimal solution, we redesign the utility function by leveraging a state space to formulate the original problem into an ordinal state based potential game, which is proved that it exists a recurrent state equilibrium point that maximizes system throughput. Furthermore, we propose two different variants of Max-logit learning algorithm based on these two games respectively: one is a simultaneous learning algorithm with less information exchange which achieves the best Nash equilibria point of the exact potential game, the other is an efficient learning algorithm for the ordinal state based potential game which can converge to the global optimization solution. Finally, numerical results are given to validate that theoretical findings. (2) We investigate the downlink outage performance of HCNs and its resource allocation problem to mitigate interferences. In HCNs, the deployed small cell base stations share the same spectrum resource with that of macro base station, which causes cross-tier interferences and the performance depreciation of the system. Specifically, we first derive closed-form expressions of overall outage probability of the system. Then, the resource allocation problem is proposed to reduce the outage probability of HCNs. This problem is formulated as a matching game, and a stable matching is considered to be the solution. Finally, to address this matching problem, a distributed algorithm is proposed which can find a stable matching. Both analytical results and simulations show the accuracy of the derivation and the overall performance improvement of HCNs by using the proposed algorithm. The numerical results finally show the validity of our analysis and that the system performance is further enhanced by using our algorithm. (3) We investigate the joint power control and subchannel allocation problem for D2D and small cell uplink communications in a cellular network, with the aim of improving the network throughput. For this throughput maximization problem, we leverage a game-theoretic learning approach to solve it. However, there is an intractable issue for the feasible region of the optimization problem, which is intermixed with the integer nature. To tackle this problem, we first present the closed-form expressions of the optimal power under different scenarios by presetting the subchannel allocation profile. Based on the optimal power control, we then formulate the subchannel allocation problem as a game theoretical framework, and define a welfare function which has the same optimum as the optimization objective. To optimize the welfare function, a distributed trial and error learning algorithm is proposed to converge to a stochastically stable state. Since the achieved stable state cannot guarantee to be the optimal solution, we reformulate this problem as an exact potential game model and propose another distributed learning algorithm to find the best Nash equilibrium which is the global optimum. Moreover, to accelerate the convergence, these two algorithms are modified by getting rid of these unavailable strategic profiles. Finally, numerical results verify that the proposed algorithms have high convergence rate with a less computation. (4) We address the downlink resource (subcarriers and power jointly) allocation problem for energy harvesting-based D2D communication in a railway carriage communication network to improve the EE of the system. The considered problem is formulated as maximizing the weighted EE and is solved by leveraging a game-theoretic learning approach. Specifically, we first propose a new performance metric for evaluating the EE and optimize its a lower bound. However, there exists an intractable issue of mixing the integer nature into the feasible region. To this end, we decompose the optimization problem into two subproblems by fixing the subcarrier and power allocations alternately. These two subproblems are formulated as two exact potential games, and the optimal properties of their solutions are analyzed. Accordingly, we respectively design a virtual distributed learning algorithm for the power control to find the global optimum (i.e., the best Nash equilibrium) based on the derived conditions of uniqueness of NE which can effectively accelerate convergence, and a fully distributed Max-logit algorithm for the subcarrier allocation to obtain the best NE with an arbitrarily high probability in which only local information needs to be exchanged. Through the alternation of two algorithms and iterative operation, the optimal solution of the problem is achieved. Finally, numerical results show that the proposed schemes have superior performance than the existing ones. (5) We tackle the problem of sum-rate maximization via joint user association and power allocation in heterogeneous cloud radio access networks while guaranteeing the QoS requirements of all users. A generalized Stackelberg game is applied to this problem with the coupled strategy set of power allocation, and a centralized-distributed method is designed to achieve the optimal solution. Specifically, user association can be efficiently solved by cloud computing and a distributed power allocation algorithm is proposed by using variational inequality theory. Finally, numerical results show that the proposed algorithm is convergent and gets a performance closed to the exhaustive method. (6) Caching popular contents in the storage of base stations has emerged as a promising solution for reducing the transmission latency and providing extra-high throughput. We tackle the optimal tradeoff problem between the sum of effective rates and backhaul saving via joint service improvement and content placement in cache-enabled heterogeneous cellular networks while guaranteeing the QoS requirements of all ground user terminals (GUTs) and backhaul traffic constraints of all base stations. However, there exists an intractable issue of mixing the integer nature into the feasible region in the nonlinear optimization problem. To this end, we decompose the optimization problem into three subproblems by alternately fixing two of three class of variables (i.e., GUT association, power control and content placement). Aiming to these subproblems, we respectively convert them into the tractable forms and propose the corresponding algorithms. By combining them, we propose a three-tier iterative algorithm for jointly optimizing GUT association and cache placement. Moreover, when GUTs have failed to be associated with ground base stations, unmanned aerial vehicle (UAV) mounted cache-enabled base stations is used as an effective solution for providing wireless services to these GUTs. The efficient deployment of UAVs (such as transmitting power, number of UAVs, locations and caching) while guaranteeing the quality of service requirements is investigated. Specifically, we propose a reinforcement learning-based approach to solve the multi-objective deployment problem while maintaining the optimal tradeoff between power consumption and backhaul saving. Finally, numerical results have verified the effectiveness of our schemes.