Simulation Study: Trajectory Tracking Control of Parafoil System Based on Deep Reinforcement Learning

Xuepu  Zhang

doi:10.6180/jase.202512_28(12).0011

Simulation Study: Trajectory Tracking Control of Parafoil System Based on Deep Reinforcement Learning

Computer Science and Information Engineering

Xuepu ZhangThis email address is being protected from spambots. You need JavaScript enabled to view it.

School of Big Data and Computer Science, Guizhou Normal University-550025, Guiyang, China

Received: October 3, 2024
Accepted: December 21, 2024
Publication Date: April 23, 2025

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202512_28(12).0011

The parafoil system is nonlinear and complex with a large time delay. This makes it challenging for traditional control methods to control the parafoil system effectively. However, the Markov property of reinforcement learning offers a new possibility for controlling the parafoil system. Therefore, this paper employs the deep reinforcement learning (DRL) method to train a neural network controller for controlling the parafoil system, based on a modified deterministic version of the distributional soft actor-critic with three refinements (DSAC T) algorithm and it is named MC-DSAC-T. The controller of the parafoil system is denoted as a multilayer perceptron (MLP) and the objective function of the policy introduces cumulative discounted rewards of a single episode to improve the stability of the iterative update of the policy, a Monte Carlo (MC) thinking. In addition, wind disturbances are introduced during training to enhance the robustness of the neural network controller. First, a nine-degree-of-freedom (nine-DOF) dynamic model of the parafoil system is developed. Secondly, the network structure of the MC-DSAC-T algorithm and the process of updating the network using sampling data were introduced. Finally, the control effects of the neural network controller trained by the proposed method were compared with those of the proportion integration differentiation (PID) controller in a wind environment. While tracking 100 randomly selected trajectory segments, the results show that the neural network controller is superior to the PID controller in distance control accuracy, which proves that the neural network controller can control the parafoil system to perform the tracking task and verify the effectiveness of the proposed method.

Keywords: Parafoil System; Deep Reinforcement Learning; Trajectory Tracking; Neural Network Controller; DSAC-T

[1] H.Zhu, Q.Sun, X. Liu, J. Liu, H. Sun, W. Wu, P. Tan, and Z. Chen, (2021) “Fluid–structure interaction-based aerodynamic modeling for flight dynamics simulation of parafoil system" Nonlinear Dynamics 104(4): 3445 3466. DOI: 10.1007/s11071-021-06486-0.
[2] Y. Li, M. Zhao, M. Yao, Q. Chen, R. Guo, T. Sun, T. Jiang, and Z. Zhao, (2020) “6-DOF Modeling and 3D Trajectory Tracking Control of a Powered Parafoil System" IEEE Access 8: 151087–151105. DOI: 10.1109/ ACCESS.2020.3016669.
[3] Q. Sun, L. Yu, Y. Zheng, J. Tao, H. Sun, M. Sun, M. Dehmer, and Z. Chen, (2022) “Trajectory tracking control of powered parafoil system based on sliding mode control in a complex environment" Aerospace Science and Technology 122: 107406. DOI: https://doi.org/10.1016/j.ast.2022.107406.
[4] T. Jann. “Aerodynamic model identification and GNC design for the parafoil-load system ALEX”. In: 16th AIAA Aerodynamic Decelerator Systems Technology Conference and Seminar. American Institute of Aeronautics and Astronautics (AIAA), 2001. DOI: 10.2514/ 6.2001-2015.eprint: https://arc.aiaa.org/doi/pdf/10.2514/6.2001-2015.
[5] Z.Zhang, Z. Zhao, and Y. Fu, (2019) “Dynamics analysis and simulation of six DOF parafoil system" Cluster Computing 22(5): 12669–12680. DOI: 10.1007/s10586 018-1720-3.
[6] N.J. Slegers, (2010) “Effects of Canopy-Payload Relative Motion on Control of Autonomous Parafoils" Journal of Guidance, Control, and Dynamics 33(1): 116–125. DOI: 10.2514/1.44564.eprint: https://doi.org/10.2514/1.44564.
[7] E. Mooij, Q. Wijnands, and B. Schat. “9 DoF parafoil/payload simulator development and validation”. In: AIAA Modeling and Simulation Technologies Conference and Exhibit. 2003, 5459. DOI: 10.2514/6. 2003-5459.eprint: https://arc.aiaa.org/doi/pdf/10.2514/6.2003-5459.
[8] O.Prakash and N. Ananthkrishnan. “Modeling and simulation of 9-DOF parafoil-payload system flight dynamics”. In: AIAA Atmospheric Flight Mechanics Conference and Exhibit. 2006, 6130. DOI: 10.2514/6. 2006-6130. eprint: https://arc.aiaa.org/doi/pdf/10.2514/6.2006-6130.
[9] Y. Gang, (2015) “Nine-degree of Freedom Modeling and Flight Dynamic Analysis of Parafoil Aerial Delivery Sys tem"ProcediaEngineering99:866–872. DOI:10.1016/j.proeng.2014.12.614.
[10] N.Slegers and M. Costello, (2005) “Model Predictive Control of A Parafoil and Payload System" Journal of Guidance, Control, and Dynamics 28(4): 816–821. DOI: 10.2514/1.12251. eprint: https://doi.org/10.2514/1.12251.
[11] J. Tao, Q. Sun, P. Tan, Z. Chen, and Y. He, (2016) “Active disturbance rejection control (ADRC)-based autonomous homing control of powered parafoils" Nonlinear Dynamics 86(3): 1461–1476. DOI: 10.1007/s11071 016-2972-1.
[12] J. Tao, Q. Sun, H. Sun, Z. Chen, M. Dehmer, and M. Sun, (2017) “Dynamic Modeling and Trajectory Tracking Control of Parafoil System in Wind Environments" IEEE/ASME Transactions on Mechatronics 22(6): 2736–2745. DOI: 10.1109/TMECH.2017.2766882.
[13] Y. Zheng, J. Tao, Q. Sun, H. Sun, Z. Chen, M. Sun, and G.Xie,(2023) “Sideslip angle estimation based active disturbance rejection 3D trajectory tracking control for powered parafoil system and hardware-in-the-loop simulation verification" Aerospace Science and Technology 141: 108497. DOI: https://doi.org/10.1016/j.ast.2023.108497.
[14] W. He, Y. Zheng, J. Wen, J. Tao, and Q. Sun, (2024) “Path Following Control of Parafoil System Based on SG LOS and Improved ADRC Tuned by MSMPA" IEEE Transactions on Circuits and Systems II: Express Briefs 71(8): 3895–3899. DOI: 10.1109/TCSII.2024.3377010.
[15] H. Sun, Q. Sun, J. Tao, S. Luo, and Z. Chen. “A flight control system for parafoils based on improved PID control approach”. In: 2017 36th Chinese Control Conference (CCC). 2017, 1168–1173. DOI: 10.23919/ChiCC.2017.8027506.
[16] J. Tao, M. Dehmer, G. Xie, and Q. Zhou, (2019) “A Generalized Predictive Control-Based Path Following Method for Parafoil Systems in Wind Environments" IEEE Access 7: 42586–42595. DOI: 10.1109/ACCESS.2019.2905632.
[17] L. Zhao, W. He, F. Lv, and W. Xiaoguang, (2020) “Trajectory Tracking Control for Parafoil Systems Based on the Model-Free Adaptive Control Method" IEEE Access 8: 152620–152636. DOI: 10.1109/ACCESS.2020.3017539.
[18] Y. P. Wang and H. R. Dong. “The Application of Adaptive PSO in PID Parameter Optimization of Unmanned Powered Parafoil”.In: Modern Technologies in Materials, Mechanics and Intelligent Systems. 1049. Advanced Materials Research. Trans Tech Publications Ltd, 2014, 1094–1097. DOI: 10.4028/www.scientific.net/AMR.1049-1050.1094.
[19] H. Jia, Q. Sun, and Z. Chen. “Application of Single Neuron LADRC in Trajectory Tracking Control of Parafoil System”. In: Proceedings of 2018 Chinese Intel ligent Systems Conference. Ed. by Y. Jia, J. Du, and W. Zhang. Singapore: Springer Singapore, 2019, 33–42. DOI: 10.1007/978-981-13-2288-4_4.
[20] Y.Zheng, J. Tao, Q. Sun, X. Zeng, H.Sun, M.Sun, and Z. Chen, (2023) “DDPG-based active disturbance rejection 3D path-following control for powered parafoil under wind disturbances" Nonlinear Dynamics 111(12): 11205–11221. DOI: 10.1007/s11071-023-08444-4.
[21] Y.Zheng, J. Tao, Q. Sun, H. Sun, M. Sun, and Z. Chen. “Path following control for powered parafoil system based on TD3-LADRC”.In: 2023 42nd Chinese Control Conference (CCC). 2023, 2364–2369. DOI: 10.23919/CCC58697.2023.10240655.
[22] Y. Zheng, J. Tao, Q. Sun, J. Yang, H. Sun, M. Sun, and Z. Chen. “Intelligent Trajectory Tracking Control of Unmanned Parafoil System Based on SAC Op timized LADRC”. In: Neural Information Processing. Ed. by B. Luo, L. Cheng, Z.-G. Wu, H. Li, and C. Li. Singapore: Springer Nature Singapore, 2024, 97–108. DOI: 10.1007/978-981-99-8073-4_8.
[23] Z. Wei and Z. Shao. “Precision landing of autonomous parafoil system via deep reinforcement learning”. In: 2024 IEEE Aerospace Conference. IEEE. 2024, 1–10. DOI: 10.1109/AERO58975.2024.10521056.
[24] Z. Wei, Y. Gao, Z. Shao, and C. Wang, (2024) “Dynamic-model-based closed-loop guidance and control for heavy parafoil system precision landing" Aerospace Science and Technology 146: 108964. DOI: 10.1016/j.ast.2024.108964.
[25] J. Duan, W. Wang, L. Xiao, J. Gao, and S. E. Li, (2023) “DSAC-T: Distributional Soft Actor-Critic with Three Re f inements" arXiv preprint arXiv:2310.05858: DOI: 10. 48550/arXiv.2310.05858. arXiv: 2310.05858 [cs.LG].