Students create photo-to-Braille translator for the blind

The display developed by students will translate any book or text from a photo into Braille 'on the fly'. The device that uses artificial intelligence can also be used by blind people to learn a language. The prototype was developed by young engineers from the Cracow University of Technology.

According to the university, the Braille display with character processing and text recognition system was developed by Bartłomiej Szostak and Hubert Orlicki at the university’s Faculty of Electrical and Computer Engineering as part of their diploma projects. The device uses image detection and alphabet teaching functionality. It is fitted with a camera that collects a real-time image of the text to be translated. With the support of algorithms, including artificial intelligence (AI), it then extends a set of pins that translate the image into Braille. The translated text is displayed from left to right on the Braille display, according to the relevant rules.

The small display box hides numerous modules, combined into an efficient mechanism. The heart of the device is a Raspberry Pi microcomputer. 'A camera and a PCA controller are connected to the microcomputer, allowing to connect the servo drives necessary to set the sliding strips in motion. The strips set the pins in the right order, which the blind person reads by touch. The OCR (optical character recognition) system uses artificial intelligence. The text recognition system is based on the Tesseract OCR engine. The device software was written in Python 3.X, with the use of OpenCV, Adafruit, and PySimpleGUI auxiliary libraries,’ says Orlicki.

The rest of the device includes cord wheels, guiding rods, sliders, cords and tensioners, i.e. mechanisms used for precise positioning of openings. They were designed in Fusion 360 and 3D printed.

The student project, costing only PLN 1,000, is already at the 6th technological readiness level (TRL). The cheap prototype of the device is already being developed to prepare it for a wider group of users. The creators want it to become a cheap device for independent use by blind people, initially intended for the Polish market. They estimate that it will take about two years. In the future, the device could be used by people all over the world - it would only have to be properly configured and equipped with additional Braille characters.

'We used only 4 characters to build a low-budget prototype and test the concept. Now we know that our idea works. Ultimately, we want to expand the keyboard to a larger number, e.g. 12-16 characters. We also want to transfer all functions (supported by an external keyboard in the prototype) into the device itself, so that a blind or visually impaired person can comfortably use it on their own,’ says the project co-author Bartłomiej Szostak.

According to Orlicki, a broader use of AI algorithms could solve the issue of image geometry accuracy: so that the text recognition engine (OCR) would be able to handle different geometry and shapes of objects, for example food products from which the image is read.

Some ideas for modifications were conceived thanks to consultations with experts from the Polish Association of the Blind. Szostak and Orlicki would now like to improve the design and ergonomics of the device, extend its capabilities so that it can read PDF formats and scan food labels. 

They are also analysing the possibility of translating websites in the form of displaying the main content in a way similar to reading newspapers. After appropriate improvements, the display would not only be useful in everyday situations, but also in libraries, schools and kindergartens, in all places where reading Braille letters is taught. In this type of facilities, for example, the function of displaying any word or letter on the device would be useful.

The scientific supervision of the students' works was carried out by Dr. Tomasz Makowski. The students received organizational and financial support from FutureLab PK. They emphasise that from the very beginning they wanted their engineering work to solve real human problems or be an introduction to research on a socially important topic.

'We decided to focus on the needs of the blind. Our project can make their everyday life easier, help in learning, studying and intellectual development,’ says Orlicki.

The authors noticed that the problem with access to literary works and textbooks concerned a huge number of people with visual disabilities - it is estimated that only 1% of the world's books and textbooks have been translated into Braille.

Meanwhile, as the students have determined, WHO research from 2019 shows that at least 2.2 billion people have been diagnosed with visual impairments, including 39 million people who have lost their sight completely as a result of diseases or injuries. Polish data (GUS) mention about 1.8 million people with diagnosed visual impairment, of which about 100,000 are completely blind. The Polish Ministry of Culture and National Heritage estimates that up to 300,000 people cannot read books printed in black print. 'Already this basic analysis of needs convinced us that we were taking up a particularly important topic,’ says Szostak.

The solution was appreciated during the Student Cyber(netics) Symposium SCS'2023, where Hubert Orlicki and Bartłomiej Szostak won the student presentations session with their display. The students are also finalists of the University Session of Scientific Clubs of the Cracow University of Technology and participants of the Scientific Session of Champions at CUT.

PAP - Science in Poland

kol/ zan/ kap/