Towards next-generation user interfaces Chinese perspective of implementing Artificial Intelligence (AI) to support people with disabilities

Abstract-

While several studies have investigated how artificial intelligence (AI) has shaped next-generation user interfaces, most of the literature is Western-based and scant information exists on the technological advancement for people with disabilities. To address this research gap, this study conducts an integrative review to show how AI in China has facilitated the development of next-generation interfaces, allowing more realistic and personalized interactions based on the different needs of people with disabilities. Specifically, it reports various cases adopted and implemented in various Chinese real-scenarios, beyond lab experiments or small tests. This study contributes to the Sustainable Development Goals (SDGs) by depicting different scenarios, techniques and interaction levels that could be adopted in various contexts to facilitate the life of people with disabilities in different fields.

Keywords- Artificial intelligence, next-generation, user interface, disability, special needs, accessibility, sustainable development goals

Introduction

The UNESCO Asia Pacific Education Bureau in 2019 and some of the World Health Organization (WHO) data stated that there are more than 1 billion people worldwide having a disability, of which 93 million are children under the age of 14 [1]. China, particularly, has 85 million people with disabilities, making it the country with the largest disabled population in the world, accounting for 6.34% of the total population. It is expected that the total number of people with disabilities in China will reach 65.3% by 2050, which will be twice that of the current total number of people with disabilities in China [2].

Meanwhile, with rapid evolution of technology, several research studies and projects have started relying on assistive technology to help people with disabilities achieve several tasks. Artificial Intelligence (AI) as a technology, particularly, has allowed developing next-generation user interfaces where various human-machine interactions are now made possible to cater to the different needs of various disability features. Human-machine interfaces are smart interfaces that provide more realistic interactions to achieve personalized human-machine interaction according to users’ abilities and pre-existing knowledge [3]. Wigdor [4] highlighted the need to investigate how emerging technologies will develop next-generation user interfaces that provide a high quality, easy to use interfaces for experts and novices alike.

The rapid technological advancement in China, especially in terms of Artificial Intelligence (AI), has attracted international attention from different fields, including economists [5], US-based political analysts [6] and security experts [7]. In 2017, the Chinese government promulgated the “Development Plan for the New Generation of Artificial Intelligence,” which proposed the guiding ideology, strategic goals, key tasks, and guarantee measures for the development of the new generation of artificial intelligence in China by 2030, deploying to build China’s leading advantage in artificial intelligence development, accelerating the construction of an innovative country and a world-leading technological power. This AI advancement has catalyzed developing next-generation interfaces generally and for people with disabilities specifically.

However, most of the literature is Western-based, and little is known on how next-generation interfaces are developed for people with disabilities in non-Western countries like China. Therefore, this study provides an integrative review on the Chinese perspective of developing next-generation interfaces. Specifically, it reports various cases adopted and implemented in various real-scenarios, beyond lab experiments or small tests. This study contributes to the Sustainable Development Goals (SDGs) by depicting different scenarios, techniques and interaction levels that could be adopted in various contexts to facilitate the life of people with disabilities in different fields.

Chinese policies to support people with disabilities

According to the State Council Information Office of the People’s Republic of China [2], the Chinese government has taken several major measures to champion the programs of persons with disabilities and improve their lives, including founding of the China Foundation for Disabled Persons in 1984, the first national sample survey on disability in 1987, founding of the China Disabled Persons’ Federation (CDPF) in 1988, and promulgation of the Law of the People’s Republic of China on the Protection of Persons with Disabilities and the first five-year program for China’s cause of these people in 1991.The State Council has released seven five-year development plans for persons with disabilities, making overall arrangements for the protection of their rights and interests. To further define the tasks and responsibility list for the protection of persons with disabilities, China has issued “Instructions of the State Council on Accelerating the Process Toward Prosperity for Persons with Disabilities”, “Outline for Accelerating the Process Toward Prosperity for Persons with Disabilities in the 13th Five-year Plan Period”, “Plan for Promoting Equal Access to Basic Public Services in the 13th Five-year Plan Period”, “The National Disability Prevention Program (2016-2020)”, and two documents of “Special Education Promotion Plan”, for 2014-2016 and 2017-2020 respectively. Launched in 2009, the three documents of the “National Human Rights Action Plan” for 2009-2010, 2011-2015, and 2016-2020 specified the tasks and goals of this mission. By April 2018, more than 80 laws and 50 administrative regulations had been passed concerning the protection of the rights and interests of persons with disabilities. In terms of assistive technologies, during the “13th Five-Year Plan” (2016-2020) period, China formulated the vision that “over 80 percent of the disabled adults (with disability certificate) and children can receive basic adaptive assistive technology products and services by 2020.” Additionally, the Ministry of Education and six other ministries jointly issued the “14th Five-Year Plan for the Development and Enhancement of Special Education” in 2021. The document states: “Strengthen the construction of digital education resources. Actively promote the digital and intelligent transformation and development of special education, explore the application of new technologies such as the internet, cloud computing, big data, virtual reality, and artificial intelligence.”

All those efforts and initiatives to support people with disabilities in line with the rapid technological advancements in China have raised the attention of people about the latest developed next-generation interfaces and how they used to cater to the different needs of disabilities, as discussed in this study.

Method

This present study was framed through an integrative literature review [8, 9]. Integrative reviews are a type of non-systematic reviews [10] aiming to cover “new or emerging topics that would benefit from a holistic conceptualization and synthesis.” While systematic reviews might inadvertently amplify knowledge inequalities by focusing on specific databases or article types [11], the integrative approach on the other hand overcomes this issue by covering knowledge in different types from different sources, regions and languages [12, 13]. This is important in this present study as the goal is to exemplify case studies of implementing AI to support people with disabilities in real contexts beyond the lab experiment context, which is usually the common practice reported in research papers [14, 15].

Our approach distinctly adopts the methodology proposed by Raffo et al. [16], known as “conceptual synthesis,” which goes beyond synthesizing substantive research findings. Instead, it aims to identify the conceptual foundations from which these findings emerge. Particularly, this present study focuses on how AI has shaped user interfaces, referred to as “next-generation interfaces”, and the different types of interaction made possible for people with disabilities and special needs to achieve various objectives (education, information access, etc.).

AI-based next-generation interfaces and interactions for people with disabilities in China

This section provides several case studies illustrating next-generation interfaces that have been implemented in various contexts to support people with disabilities.

• Accessible information search and retrieval

China’s leading search engine provider Baidu has announced the Blind Search device (see Figure 1), a tool to assist blind and vision impaired people access “massive amounts of information online through touch” using a combination of tactile and voice-activated commands. The Blind Search tool provides a dedicated area for typing commands using finger gestures, which are converted to Braille output via a display at the top of the device so that the users can check if there are any typing errors. Once search terms have been entered, a button at the back of the device can be pressed to display a list of text-based search results. These results are displayed in Braille output to be read by users or they are also converted to audio through text-to-speech functionality. Voice input is also supported, allowing users to press a button at the side of the device to speak search terms aloud, which are then converted into text or image-based results via the tactile display.

Images can also be searched and displayed using a “tangible image made from dot positions arranged in rectangles” which the user can interact with via touch, providing audio description as users navigate across different sections of the picture. As Baidu states, the combination of providing tactile imagery with audio description is intended to “help blind people understand [the] search results in a comprehensive stereoscopic way”.

Figure 1. Blind Search device [17]

• Easy communication and interaction

Special Education College of Beijing Union University is the only college in Beijing that focuses on higher education for persons with disabilities. At present, 70% of the students in the college are students with disabilities, and 70% of the students with disabilities are hearing-impaired. The school uses an AI-based smart classroom, where teachers’ speeches in classrooms are automatically translated into subtitles and sign-languages and then displayed on a large screen in classes (see Figure 2).

Figure 2. Art and Design Professional Teacher is teaching to Disabled Students [18]

For blind people, the school has also developed and equipped students with reading pens which are used to scan the letters and these letters are then read out loud (see Figure 3). This tool allows self-studying and avoids issues, including having only four or five books for Braille, which is very inconvenient to check; the second obstacle is that the information they master through touch is sometimes incomplete, and they need to turn over the books to form a connection with the things they touch. This pen can also be used to read images which are converted into 2D convex images.

Figure 3. Diagram of the Voice QR Code Reading pen on the medicine cabinet [18]

• Cognitive Development in immersive virtual reality (VR) environments

Shunde District Qizhi School is a comprehensive boarding special education school for children with moderate to severe intellectual disabilities. There are VR labs to develop students’ cognition, overcome limitations and monotonous methods in safety education, as well as issues related to the difficulty and high cost of setting up training scenarios (see Figure 4). For instance, abstract concepts like ‘up-down’ and ‘left-right’ are traditionally taught using a point of reference. However, for children with intellectual disabilities, they might struggle to understand these concepts when the environment changes, yet real-life teaching cannot provide a variety of scenarios. VR can simulate numerous scenarios for intensive student training, enhancing their understanding of abstract concepts through continuous stimulation and fostering the ability to generalize these concepts.

Take safety education as another example, real-life teaching cannot allow students to touch fire or experience the pain of drowning by jumping into water. VR simulates such scenarios, and when students enter those virtual environments, the system triggers alarms or produces smoke signals to indicate danger. Through repeated reinforcement, students become aware of potential hazards, achieving the goal of training their safety awareness. Teachers can also monitor the virtual scenarios experienced by students in real-time on a pad device, providing timely instructional guidance and intervention. Students also receive feedback within the VR environment; for instance, if they perform well, the computer may emit an encouraging sound saying, “You did great!”

Figure 4. Special education students learn through VR [19]

• Early diagnosis and intervention for children with Autism

In March 2024, the “China Autism Education and Rehabilitation Industry Development Status Report” was officially released. The report estimates that the prevalence of autism among children aged 6-12 is 0.7%. Conservatively estimated, there are about 2 million children with autism aged 0-14 nationwide, with an annual increase of approximately 160,000. Professor Xu Peng from the University of Electronic Science and Technology of China has combined artificial intelligence techniques such as deep learning, transfer learning, and manifold learning to establish an early diagnostic recognition system for children with ASD (Autism Spectrum Disorder), focusing on the abnormal interaction information between brain networks and other source signal networks [20]. This system improves the accuracy of diagnosis and helps patients with autism receive early intervention, achieving better outcomes in lifelong rehabilitation.

Currently, intervention technologies for autism face issues such as a lack of personalized intervention and insufficient levels of intelligence. Over the past decade, Professor Chen LiangYing’s team at Central China Normal University has been continuously tackling these challenges, focusing on refined assessment and personalized intelligent intervention for children with autism. Her research has explored intelligent recognition methods for children with autism that integrate multimodal data. By analyzing behavioral and cognitive data, significant differences were found between children with autism and typically developing children in eye movements, facial expressions, cognitive scores, and cognitive response time data. Furthermore, feature selection was performed using these data differences, and an autism recognition model integrating multimodal data was constructed. Figure 5 shows the data from a child with autism are being collected through a head-mounted device while he is playing games.

Figure 5. A child with autism is being assessed [21].

• Intelligent sports equipment aids in enhancing the physical fitness of special-needs children

Since most children with disabilities are unable to express their adverse physical reactions normally using smart teaching devices such as jump ropes and wristbands to monitor their heart rate and other physical indicators can effectively prevent the occurrence of safety accidents. These intelligent devices provide teachers with a better understanding of students’ physical conditions and offer precise exercise data to physical education teachers, allowing them to develop personalized teaching plans for students with different physical disabilities.

Xiangyang Special Education School has a total of 14 teaching classes with 180 students enrolled. In 2022, the school launched a smart physical education project, including a smart physical education platform and intelligent devices, which have now been fully implemented in each class. The school also conducted a comparative experiment on the one-minute rope skipping event, showing that the overall average score of students in the experimental group (using the smart physical system) improved by 20.3%. At the same time, the teachers stated that “the smart physical system has made the physical education teaching process quantifiable and visual, promoting classroom communication between teachers and deaf students or students with intellectual disabilities, effectively enhancing the quality of physical education teaching.” Figure 6 shows the data on a student’s psychological baseline indicators, average heart rate curve, and exercise intensity index after exercising.

Figure 6. Student’s psychological data [22]

• Empower special education teachers through generative AI

On November 9, 2023, the Tianhe District Education Bureau in Guangzhou held a seminar on the exploration and practice of AI+ Special Education Practice Community. A teacher from Qihui School in Tianhe District presented an open class demonstration titled “Carmen Overture Music Appreciation – Rhythm.” Before the class, the teacher used generative artificial intelligence to create personalized learning objectives and strategies for eight children with varying degrees of intellectual disabilities. Figure 7 shows the generative AI system, iFlyTEK Spark, responds to the following prompt:

“You are an esteemed professor specializing in special education. I have eight children with various disabilities, including three with intellectual disabilities, two with Down syndrome, two with autism, and one with cerebral palsy. Classified by the degree of impairment: one mild, two moderate, three severe, and two profound. Please provide an analysis of their learning situations when teaching ‘Carmen Overture,’ focusing on three aspects: musical perception ability, rhythmic expression ability, and social adaptation skills. The analysis should be presented in a tabular form for each type of child.”

Figure 7. Response of the generative AI tool

It should be noted that iFLYTEK Spark is a general-purpose large language model and is not specifically designed for the field of special education. In China, there are also GAIs that specifically serve special education teachers, such as the one shown in Figure 8, namely QiWei special education AI partner. It can provide teachers with functions like lesson plan writing, rehabilitation training activity design, special education encyclopedias, and special education event planning assistants and so on.

Figure 8. The webpage of QiWei special education AI partner [23]

Conclusion and future directions

This study conducted a review to highlight how emerging technologies like AI in China have sharpened next-generation interfaces. The obtained results revealed that Internet of Things (IoT) and immersive technologies like AR have helped to provide more natural interactions, where multimodal data are collected and analyzed in immersiveness environments stimulating real-world scenarios. This reveals that such technologies can provide new types of interaction beyond the traditional ones which usually relies on clicking (through mouse) and typing (through keyboard). It is also seen that China has relied on portable and open source technologies to cater to the different needs of people with disabilities. In this context, several studies highlighted the importance of harnessing the power of “openness” to promote accessibility and inclusivity [24, 25].

Additionally, it is seen that several case studies have relied on large language models (LLM) and generative AI (Gen AI) to generate different solutions (content, teaching strategies, etc.) for people with disabilities based on their needs. While Gen AI is a great technology to support inclusivity and accessibility, Tlili et al. [26] also pointed out the need to keep a close eye on the ethical issues of such technology as it might harm users instead of helping them.

This study also revealed the close relationship between research and development (R&D) in China which facilitated the rapid development of next-generation interfaces for people with disabilities. This was seen where various leading industries and companies have closely worked with special education schools to develop low-cost and portable assistive technologies.

References

• UNESCO Asia Pacific Bureau of education. (2019). Focusing on “hidden disabled” children, world education information, 2019, 11.
• State Council Information Office of the People’s Republic of China. (2019). Equality, Participation and Sharing: 70 Years of Protecting the Rights and Interests of Persons with Disabilities in the PRC. Available on: http://www.scio.gov.cn/zfbps/ndhf/2019n/202207/t20220704_130618.html
• Jacob, R. J. (2006). What is the next generation of human-computer interaction?. In CHI’06 Extended Abstracts on Human Factors in Computing Systems (pp. 1707-1710).
• Wigdor, D. (2010). Architecting next-generation user interfaces. In Proceedings of the International Conference on Advanced Visual Interfaces (pp. 16-22).
• Barton, D., Woetzel, J., Seong, J., Tian, Q. (2017) Artificial intelligence: implications for China. McKinsey Global Institute, San Francisco Beijing AI Principles (2019) Beijing Academy of Artificial Intelligence. https ://www.baai.ac.cn/blog/beiji ng-ai-principles
• Kempe, F. (2019). The US is falling behind China in crucial race for AI dominance. CNBC. Accessed April 02 2024. https://www.cnbc.com/2019/01/25/chinas-upper-hand-in-ai-racecould-be-a-devastating-blow-to-the-west.html
• Allen, G. C. (2019). Understanding China’s AI Strategy: Clues to Chinese Strategic Thinking on Artificial Intelligence and National Security. Center for a New American Security. http://www.jstor.org/stable/resrep20446
• Salha, S., Tlili, A., Shehata, B., Zhang, X., Endris, A., Arar, K., Mishra, S., & Jemni, M. (2024). How to Maintain Education During Wars? An Integrative Approach to Ensure the Right to Education. Open Praxis, 16(2), 160–179. DOI: https://doi.org/10.55982/openpraxis.16.2.668
• Torraco, R. J. (2005). Writing Integrative Literature Reviews: Guidelines and Examples. Human Resource Development Review, 4(3), 356–667. DOI: https://doi.org/10.1177/1534484305278283
• Souza, M. T. D., Silva, M. D. D., & Carvalho, R. D. (2010). Integrative review: what is it? How to do it?. Einstein, 8, 102–206. DOI: https://doi.org/10.1590/s1679-45082010rw1134
• Atenas, J., Nerantzi, C., & Bussu, A. (2023). A conceptual approach to transform and enhance academic mentorship: Through open educational practices. Open Praxis, 15(4), 271–287. DOI: https://doi.org/10.55982/openpraxis.15.4.595
• Almeida, C. P. B. D., & Goulart, B. N. G. D. (2017). How to avoid bias in systematic reviews of observational studies. Revista CEFAC, 19, 551–155. DOI: https://doi.org/10.1590/1982-021620171941117
• Kordzadeh, N., & Ghasemaghaei, M. (2022). Algorithmic bias: Review, synthesis, and future research directions. European Journal of Information Systems, 31(3), 388–809. DOI: https://doi.org/10.1080/0960085X.2021.1927212
• Tlili, A., Denden, M., Duan, A., Padilla-Zea, N., Huang, R., Sun, T., & Burgos, D. (2022). Game-based learning for learners with disabilities—What is next? A systematic literature review from the activity theory perspective. Frontiers in Psychology, 12, 814691. https://doi.org/10.3389/fpsyg.2021.814691
• Tlili, A., Lin, V., Chen, N. S., & Huang, R. (2020). A systematic review on robot-assisted special education from the activity theory perspective. Educational Technology & Society, 23(3), 95-109.
• Raffo, C., Dyson, A., Gunter, H., Hall, D., Jones, L., & Kalambouka, A. (2007). Education and poverty: A critical review of theory, policy and practice. York: Joseph Rowntree Foundation.
• Media Access Australia. (2015). Chinese Tech Giant BAIDU Announces Blind Search Device. Retrieved from https://mediaaccess.org.au/latest_news/web/chinese-tech-giant-baidu-announces-blind-search-device.
• Guo, Y. (2019). AI Empowers Special Education: Enabling Deaf Students to “Hear” and Blind Students to “See”. Retrieved from https://baijiahao.baidu.com/s?id=1633649241615557222&wfr=spider&for=pc.
• Mo, S. (2023). When Special Education Meets VR Technology, a Group of Special Children in Shunde Entered a New World. Retrieved from https://content.foshanplus.com/newsDetails.html?newsId=768765
• Tian, Y. (2024). How Can AI Help Children with Autism? An Exclusive Interview with the Director of Chengdu Frontier Brain-like Artificial Intelligence Innovation Center. Retrieved from https://baijiahao.baidu.com/s?id=1795192308285836396&wfr=spider&for=pc
• China Education News Network. (2023). 2023 New Era Teacher Style | Chen Liangying: Because of AI, So Love. Available on: http://www.jyb.cn/rmtzcg/xwy/wzxw/202309/t20230906_2111088881.html
• Liu, H., & Gong, Y. (2023). Intelligent Physical Education Classes Support the Development of Sports Specialties in Special Education Schools. Retrieved from https://news.hbtv.com.cn/p/2436020.html
• Shanghai Qiwei Information Technology Co., Ltd. (2023). Welcome to QiWei special education AI partner. Retrieved from https://www.tejiao123.com/#/
• Tlili, A., Altinay, F., Huang, R., Altinay, Z., Olivier, J., Mishra, S., Jemni, M., & Burgos, D. (2022). Are we there yet? A systematic literature review of Open Educational Resources in Africa: A combined content and bibliometric analysis. Plos one, 17(1), e0262615. https://doi.org/10.1371/journal.pone.0262615
• Zhang, X., Tlili, A., Nascimbeni, F., Burgos, D., Huang, R., Chang, T. W., Jemni, M., & Khribi, M. K. (2020). Accessibility within open educational resources and practices for disabled learners: A systematic literature review. Smart Learning Environments, 7, 1-19. https://doi.org/10.1186/s40561-019-0113-2
• Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x