Low-Cost Electronic Prosthesis with Object Detection
ABSTRACT
Users of motorized prosthetic limb may encounter difficulties when trying to grip objects. When grabbing objects, users need to change the mode (pinching, semi-pinching and gripping) depending on the object size. The solution found for this problem was to add an Artificial Intelligence (AI) to the prosthesis so that it can automatically recognize which mode to use. The AI has real time object detection which allows for this advancement. This system is connected with a controller of the hand so it executes the actions for the user when they recognize registered objects. Furthermore, research has found that it is becoming increasingly difficult to find a medical professional able to fit a prosthetic and it is also expensive to purchase one. [1] [2] Therefore, by building a prosthetic hand using a 3D printer, users can benefit from an easier use as the AI makes actions easily executable and it is a more time and cost-efficient option.
INTRODUCTION.
Most practical prosthesis cost over $5,000 [1]. The advantages to building a prosthesis using a 3D printer include: a lower production price, less aesthetic work is needed (such as welding and coloring), and using plastic makes the prosthesis lighter. Furthermore, the user does not need to wait for the product dispatchment which allows the process of building to be faster. Currently, most of the prosthesis that are available to buy on the internet are made for robot use rather than human use.
In 2010, 219 prosthesis were ordered in Japan. The myoelectric prosthesis (a prosthetic hand which moves by sensing muscle electricity) constituted 2% of overall orders. The report stated, the main reason only 2% of the myoelectric prosthesis were being ordered was due to the fact that the amputees need to have a specific reason (such as educational needs) to acquire a model superior to that of the standard active artificial hand [2].
The main reasons for amputation are congenital, tumor, disease, and trauma. The most common reason for amputation is trauma constituting of 77% of amputations. Despite 50,000 new amputations every year in the United States of America, only half of all upper extremity amputees received prosthetic services. [3]
The short-term goal for this project was to create a low-cost prosthetic hand with AI object detection for under $100. Following this, the ultimate aim of this project is to support amputees, giving them more available prosthesis options to make their lives easier.
MATERIALS AND METHODS.
To build this prosthetic (Figure 1), 8 prototypes were made. All the parts were designed using 3D Computer-aided design (CAD) software.
Figure 1. Overview of the prosthetic with AI. (A) Basic components need to build a thumb. Overview of the (B) built thumb and (C) whole prosthetic.
Embedded System (Mainboard)
The simplest way to build the fingers was to use micro servos (SG90). A wireless transceiver (NRFL01+) was used to communicate with a remote controller. All electronics are controlled by a micro-controller (Arduino nano). Finally, the hand has three lithium ion batteries to produce 12V to control the servos.
Object detection software
To process the execution of each movement, object detection software was added (Chart S1). This program works by editing (changing colors and sizes) a live feed from the camera and converting the image to a numerical array. Then each block of the image will be fitted to the tensors and will export the weight of accuracy. The accuracy is determined using Equation 1, where ‘Actual’ is the tensor output of MVNC and ‘Expected’ is the tensor output of TensorFlow.
\[Accuracy=\frac{Max(\left|Actual-Expected\right|)\ }{Max\left(\left|Expected\right|\right)}\tag{1}\]
Embedded System (Radio Module)
The radio controller includes a gyro sensor (MPU6050), radio module (NRFL01+), and a pressure sensor (HbSensorBoard01). The gyro sensor is used to change the modes of grip. The pressure sensor reads the value of the muscle pressure and transmits the data using the radio module to the main board. Arduino nano is used in the remote controller to make its size compact.
Underactuated mechanism
When testing the grip of the fingers, it was found that they could grab any objects that was designed to be lifted (such as a smartphone). However, there was no friction and therefore a smartphone would slip from the grip because the fingertips were moved using gears which created a gap between the object and the hand. The solution was to use the Underactuated Mechanism (Figure 2A). This mechanism gives the fingers a variety of angles to bend at. As figure 2C shows, this mechanism minimizes the use of servos in each finger, which also makes the hand more cost efficient (Figure 2B) (Figure 3).
Figure 2. Basic information about each finger with underactuated mechanism. (A) The finger joints and their respective lengths. (B) The way that the servo is mounted to the finger. (C) Overview of the whole finger and how it bends.
Figure 3. The underactuated mechanism allows the finger to bend in a variety of ways so the fingers can fit the object much more accurately.
A human thumb is predominantly used to assist the movement of four fingers. Consequently, the thumb does not have an underactuated mechanism, but instead it has a geared mechanism so it can change the position for pinching an object (Figure 1A & 1B).
Servo
When testing the finger abilities of the hand, the servos heat up and break if the hand is used for over two minutes continuously. Therefore, an Optimal Pressure Manager (OPM) system was installed to read the voltage of the servos while the fingers are retracting and stops at the position when the voltage drops due to the resistance caused by the object. Also, this system increases the duration of the battery by reducing the power loss to the servos while it is trying to add force (Chart S2).
RESULTS.
An experiment was conducted in order to test the accuracy of the object detection software. It was found that the software recognized common objects that are used in daily life such as a laptop, a ball point pen, and a coffee mug. As table 1 shows, the average recognition accuracy was 63% and the average time to recognize an object was five seconds.
Table 1. Table of the experiment of average recognition accuracy and time it took.
To improve the recognition accuracy, instead of using pre-trained object detection, deep learning could be used as it automatically updates as it recognizes an object. This could be possible by collecting images and creating a new dataset and therefore, increase the number of recognizable objects.
In Figure 4, the program recognized a water bottle. The object detection program was found to detect an object at around 15 frames per second.
Figure 4. Hand recognizing a Pet bottle using AI program.
Hand is recognizing the bottle by outputting the name of object on the first row and the second row shows the mode of gripping for the object that was detected which are g (grip), s (semi-pinch), and p (pinch).
DISCUSSION.
As previously stated, one of the current problems with standard prosthesis is the need to perform additional actions (such as pressing buttons). The solution to this was adding object detection software which controls the hand by recognizing the object it will pick up. However, prosthetics remain expensive and difficult to procure.
Overall, the prototype built during this project resulted in a more cost efficient and accessible option for amputees. Similarly, the object detection software provided should improve the ease of use for amputees.
However, there are still limitations that need to be addressed. For example, if the servos was damaged, it would take time to be replaced. Similarly, the object detection software can only recognize specific objects that it was pre-trained to recognize, which limits its use. Therefore, the future goal would be to provide all users with extra parts (particularly for fragile components) when purchased or to improve the quality of the parts so they will not fail. To address the object detection limitations, a deep learning software would be needed to detect a new object and add the object to its database.
By building a database server and sending the recognized images and object data to a cloud, the server would automatically update the dataset in the hand and also data can be shared with other users of these hands, which can be beneficial as learning is made even faster. Eventually, the server will contain a bank of object data. Going forward, the intention would be to make the deep learning program and the hardware required to use the program open source, so it could be used with a preferred prosthesis model.
CONCLUSION.
Overall, this prosthesis is an alternative option that was cheaper to create, durable, and could recognize an object and select a more efficient way to grab it. Upon further research, a deep learning engine, collecting the images generated from the hand could be used. This would analyze the images and create a dataset to improve the accuracy of the recognition. Furthermore, this prototype and its capabilities could be tested out by asking for volunteers for feedback on the ease of use.
ACKNOWLEDGMENTS.
I would like to thank Mr. Schwartz and Ms. Moran for proofreading this essay. Other thanks go to Mrs. Conrad for finding this journal website.
SUPPORTING INFORMATION.
Chart S1. Visual map of the structure of the electronic system of the prosthetic.
Chart S2. Description of the OPM system and how it works.
REFERENCES.
- Sharington, Grace. “The True Cost of Prosthetic Limbs: What It Means For Families.” Jordan Thomas Foundation, 29 June 2017, jordanthomasfoundation.org/true-cost-prosthetic-limbs/.
- Kashimoto, Shuu. “Payment and Tasks of Myoelectric Wristbands in the Independence Support Law for Persons with Disabilities.” Payment and Tasks of Myoelectric Wristbands in the Independence Support Law for Persons with Disabilities., 22 Apr. 2013, www.jsomt.jp/journal/pdf/061050305.pdf.
- Kulley, Marlowe. Hand Prosthetic. 2003, Hand Prosthetic, web.archive.org/web/20151217151109/http://biomed.brown.edu:80/Courses/BI108/BI108_2003_Groups/Hand_Prosthetics/stats.html.
Posted by John Lee on Wednesday, December 23, 2020 in May 2019.
Tags: 3D printed prosthesis, AI, Artificial Intelligence, Low-cost prosthesis, Object Detection