Train the RASAT model that integrates the relational structure into the pre-trained Seq2Seq model to convert text into SQL

Abstract


In this study the relational attention module is getting integrated into pre-trained Transformer Seq2Seq model and realize the conversion of natural language questions to Structured Query Language (SQL) retrieval commands by conducting experiments on the Spider dataset. The purpose of this scientific article is to improve the accuracy and efficiency of converting text into SQL queries by using the relational attention mechanism in the transformer model. The article presents the RASAT (Relational Attention-based SQL Transformation) model, which replaces the self-rotation module in the transformer encoder with a relational attention module for processing text-to-SQL tasks. this approach allows you to better take into account the semantic relationships between entities in the text and generate more accurate SQL queries. The research methods include the use of a pre-trained transformer model (T5-small) and training it on the Spider dataset with the introduction of a relational attention module. Experimental results show a significant improvement in accuracy indicators when converting text to SQL compared to the basic model without a relational component. The experimental results demonstrate that the RASAT model improves the Exact Match performance by 1.82% and the Execution Accuracy by 3.26%. These improvements are achieved despite the fact that the number of training epochs was limited to 500 instead of 3072 for the basic model, which emphasizes the effectiveness of the proposed approach even with limited computing resources. In conclusion, the prospects for further development of the relational model method to improve the quality of systems related to natural language processing and databases are emphasized.

Full Text

5

About the authors

Lai Xifei

Novosibirsk State University

References

  1. Kamath, A. A Survey on Semantic Parsing / A. Kamath, R. Das // arXiv. – 2018. – 1812.00978. – doi: 10.48550/arXiv.1812.00978
  2. Xu, X. SQLnet: Generating Structured Queries from Natural Language without Reinforcement Learning / X. Xu, C. Liu, D. Song // arXiv. – 2017. – 1711.04436. – doi: 10.48550/arXiv.1711.04436
  3. A Comprehensive Exploration on wikiSQL with Table-aware Word Contextualization / W. Hwang, J. Yim, S. Park, M. Seo // arXiv. – 2019. – 1902.01069. – DOI 10.48550/ arXiv.1902.01069
  4. RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases / D.H. Choi, M. Ch. Shin, E.G. Kim, D. R. Shin // Computational Linguistics. – 2021. – Vol. 47, № 2. – P. 309–332. – doi: 10.1162/coli_a_00403
  5. Bogin, B. Global Reasoning over Database Structures for Text to-SQL Parsing / B. Bogin, M. Gardner, J. Berant // Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), 3–7 November 2019, Hong Kong, China. – Association for Computational Linguistics, 2019. – P. 3659–3664. – doi: 10.18653/v1/D19-1378
  6. LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations / R. Cao, L. Chen, Z. Chen, Y. Zhao, S. Zhu, K. Yu // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 1–6 August 2021, Online. – Association for Computational Linguistics, 2021. – P. 2541–2555. – doi: 10.18653/v1/2021.acl-long.198
  7. SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL / R. Cai, J. Yuan, B.Xu, Z. Hao // arXiv. – 2021. – 2111.00653. – doi: 10.48550/arXiv.2111.00653
  8. S2SQL: Injecting syntax to question-schema interaction graph encoder for text-to-SQL parsers / B. Hui, R. Geng, L. Wang, B. Qin, B. Li, J. Sun, Y. Li // arXiv. – 2022. – 2408.03256. – doi: 10.48550/arXiv.2408.03256
  9. Relational Graph Attention Network for Aspect-based Sentiment Analysis / K. Wang, W. Shen, Y. Yang, X. Quan, R. Wang // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5–10 July 2020, Online. – Association for Computational Linguistics, 2020. – P. 3229–3238. – doi: 10.18653/v1/2020.acl-main.295
  10. Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? / P. Shaw, M.-W. Chang, P. Pasupat, K. Toutanova // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 1–6 August 2021, Online – Association for Computational Linguistics, 2021. – P. 922–938. – doi: 10.18653/v1/ 2021.acl-long.75
  11. Scholak, T. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models / T. Scholak, N. Schucher, D. Bahdanau // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, 7–11 November 2011. – Association for Computational Linguistics, 2021. – P. 9895–9901 – doi: 10.18653/v1/2021.emnlp-main.779
  12. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task / T. Yu, R. Zhang, K. Yang [et al.] // Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, 31 October – 4 November 2018. – Association for Computational Linguistics, 2018. – P. 3911–3921. – doi: 10.18653/v1/D18-1425
  13. Shaw, P. Self-Attention with Relative Position Representations / P. Shaw, J. Uszkoreit, A. Vaswani // Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, 1–6 June 2018. – Association for Computational Linguistics, 2018. – P. 464–468. – doi: 10.18653/v1/N18-2074.
  14. Attention is All you Need / V. Ashish, N. M. Shazeer, N. Parmar [et al.] // Advances in Neural Information Processing Systems. – 2017. – Vol. 30. – 11 p.
  15. UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models / T. Xie, C. H. Wu, P. Shi [et al.] // arXiv. – 2022. – 2201.05966. – doi: 10.48550/arXiv.2201.05966
  16. Yale Semantic Parsing and Text-to-SQL Challenge: сайт [Электронный ресурс]. – URL: https://yale-lily.github.io/spider (accessed 15.11.2024)
  17. RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL / J. Qi, J. Tang, Z. He [et al.] // arXiv. – 2022. – 2205.06983. – doi: 10.48550/arXiv. 2205.06983
  18. Giordani, A. Corpora for Automatically Learning to Map Natural Language Questions into SQL Queries. / A. Giordani, A. Moschitti // Proceedings of the International Conference on Language Resources and Evaluation, LREC 2010, 17–23 May 2010, Valletta, Malta. – European Language Resources Association (ELRA), 2010. – P. 2336–2339.

Statistics

Views

Abstract - 47

PDF (Russian) - 8

Refbacks

  • There are currently no refbacks.

This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies