ANALYSIS OF FATIGUE AMONG BAGGAGE HANDLERS USING THE FACIAL ACTION CODING SYSTEM (FACS)

Baggage handlers in the informal sector are individuals who provide transportation and heavy lifting services to earn a livelihood. However, this practice often leads to exceeding the recommended capacity when transporting goods. To visually analyze facial movements and detect fatigue, the Facial Action Coding System (FACS) was used to provide a comprehensive anatomical description. Based on this condition, this research aimed to detect fatigue-indicating Action Units (AUs) in the facial expressions of baggage handlers and also analyze their level of fatigue. The results showed that there were significant changes in the value of AUs, indicating fatigue in the facial expression of workers before and after work. Specifically, there was an increase in the values of AU 01, AU 15, AU 20, and AU 23, while AU 12, which was associated with smiling expressions, exhibited a decrease in facial expressions among the subjects, as supported by the results of the paired sample t-test. This research were expected to significantly impact the sustainability of baggage handlers’ employment. This impact was particularly evident to increase income and performance decline caused by fatigue. Furthermore, this research could play a vital role in preventing work accidents resulting from the excessive carrying of loads.


INTRODUCTION
Baggage handlers are individuals who work in the informal sector by offering services to transport goods or materials from one place to another.These workers generally engage in manual labor, such as carrying items, and are commonly found in areas with high economic activities, namely ports and other facilities that require transportation services.In particular, workers at Yos Sudarso Port in Ambon are part of a cooperative organization called the Unloading Workers (TKBM).
On a typical day, baggage handlers at Yos Sudarso Ambon Port work for approximately 1-2 hours.Recent surveys indicate that there are around 440 workers at the port, with 50 assigned to each transporting session.During a session, a worker can transport goods back and forth up to 10 times within 2 hours.The amount of work performed determines their daily.Consequently, baggage handlers often compete to transport more goods in order to increase their earnings.This situation causes many workers to carry loads beyond their capacity, which can lead to subjective feelings.
These feelings include facial fatigue, dizziness, sluggish thinking, reduced alertness, slow and poor perception, reluctance to work, and decreased physical and mental performance [1].It is crucial to acknowledge that these feelings are often recognized in the facial expressions of baggage handlers [2].Research investigated facial expressions as indicators of fatigue levels, such as real-time detection of driver fatigue [3] and identifying student fatigue in distance learning [4].However, research on fatigue levels among workers has not been carried out.For this reason, this current research aims to analyze fatigue using the Facial Action Coding System (FACS) based on the Computer Expression Recognition Toolbox (CERT).
Dr. Paul Ekman, a psychologist, conducted research showing that facial expressions are universally understood.This is evident from the similarity in facial expressions among residents of the United States (US) and Papua New Guinea, who have not seen television programs that influence their expressions [5].Recognizing signs of fatigue through workrelated facial expressions is an important research focus.Since the FACS can identify fatigue, the current research seeks to use the system automation on CERT in order to analyze the facial expressions of Fatigued workers.
The primary objective of this research is twofold, firstly, it aims to identify the specific Action Units (AUs) that indicate fatigue in workers.Secondly, it seeks to analyze the percentage of fatigue levels in workers before and after work.This research was conducted to evaluate the differences in the values of AUs that indicate fatigue in baggage handlers before and after work using paired sample t-tests.The significance of this investigation lies in its contribution to the field of industrial psychology within organizations, using artificial intelligence tools.Specifically, this research addresses the challenge faced by workers, providing a fresh perspective and comprehensive insights, compared to existing ones.The results are expected to have a significant impact on the sustainability of baggage handler performance, particularly at Yos Sudarso Ambon Port, by increasing income and mitigating performance decline due to fatigue, illness, or work accidents resulting from excessive load carrying.
The research content is organized into several sections based on its objectives.The introduction provides background information, and the literature review discusses previous works related to FACS and CERT.The method section describes the data processing and analysis methods, while the results and discussion was contained in the fourth section.Finally, the conclusion summarizes the results and provides recommendations for future works.

Literature Review
Fatigue is a known factor that contributes to human stress [6], and can be identified through facial expressions, such as in motorized vehicle drivers [7].Analyzing human facial images reveals a variety of expressions that offer insights into the psychological state of an individual [8].According to Paul Ekman, facial expressions comprise 6 basic emotions and 46 human facial codes [9].While some expressions and codes can be easily discerned, others often present challenges [10].Technology advancements have facilitated the automatic detection of human facial expressions.For example, Littlewort et al. [11] developed CERT, which utilizes basic emotions and FACS principles established by Paul Ekman.CERT is very useful for detecting various facial expressions, enabling informed decision-making.Additionally, its applications range from determining the needs of individuals with autism [12], detecting deception [13], and identifying signs of pain [14].
The identification of human facial expressions, particularly through FACS, is increasingly utilized.An example of this is identifying facial muscle fatigue, which has implications for understanding human stress levels.Marco C. Uchida [15] conducted research on fatigue among gymnasium athletes while lifting loads of varying weights.FACS is also used to identify fatigue, in different contexts, such as detecting driver fatigue based on eye blinking patterns [16] and identifying student fatigue in distance education settings [4].

FACS
FACS is a comprehensive system based on facial anatomy that provides a visual description of all observable facial movements.Furthermore, it breaks down facial expressions into individual muscle movements, called AUs.FACS consists of 46 AUs that describe basic facial movements by analyzing muscle activity and detailing the influence of each AU on facial features [5].

CERT
CERT is an end-to-end system designed for real-time, fully automated facial expression recognition [17].The system detects frontal faces in video streams automatically and encodes each frame using 40 continuous dimensions, including the basic expressions of anger, disgust, fear, joy, sadness, surprise, multiple head poses, and 42 AUs from FACS.It is important to note that the technical approach to CERT is an appearance-based discriminatory method.Current research focuses on machine learning and computer vision method to enhance facial expression detection, dynamics analysis, and temporal segmentation [11].CERT interface (adopted from [18])

Data Visualization
Data visualization serves the function of effectively presenting and communicating information through visual representations of data.It is not necessary for data visualization to be boring to be functional or overly sophisticated to be visually appealing.To convey ideas effectively, a balance between aesthetic form and functionality is crucial, as it provides insights into complex and sparse datasets by intuitively communicating their key aspects.Unfortunately, designers often overlook this balance, creating visually stunning data visualizations that fail to fulfill their primary purpose [19].

RESEARCH METHODS
Knowledge Discovery in Databases (KDD) was a complex nontrivial process aimed at discovering and identifying patterns valid, novel, useful, and understandable patterns in data.Furthermore, it encompassed engineering integration, scientific discovery, interpretation, and visualization of patterns across multiple datasets.
Previous works utilized Google Colab and Python to facilitate the processing of large datasets.Moreover, in this research, the knowledge data discovery process involved several stages.The first step was data collection, followed by data preparation, processing, cleaning, visualization, and finally, data analysis and conclusions.

Data Collection
During the data collection stage, video data of baggage handlers who have worked for 2 hours, were gathered.This duration was allocated to each worker, regardless of the weight of the load.The video data were captured using a Samsung Galaxy M21 mobile phone camera with a video resolution of 1080p: 1920x1080.All workers included in this research were male with the age range of 25 to 55 years.A total of 50 workers were recorded in the videos, both before and after carrying the load.

Data Preparation
The data preparation stage involved editing the baggage videos using the VSDC Video Editor application.The aim was to create 5second video clips with the faces of subjects fully facing the camera.This was to ensure proper capture by CERT, and in total, 100 videos (before and after carrying the load) were obtained, each with a duration of 5 seconds [20].

Data Processing
The data processing stage commenced by inputting video data into the CERT application to read FACS.Subsequently, FACS presentation data were obtained from each stimulus, generating datasets for 46 AUs, with 5 read across 100 videos.

Data Cleaning
The data cleaning stage involved implementing cleaning techniques on the previously processed data using Python.This included removing missing values, and outliers, and making other necessary adjustments to ensure a clean dataset was obtained, ready for the next stage.

Data Visualization
In the data visualization stage, the Matplotlib library in Python was utilized to visualize and explore AU 1, AU 12, AU 15, AU 20, and AU 23.This facilitated the creation of graphs and presentation values for these specific AUs.

Data Analysis
In the data analysis stage, descriptive statistical values of AU 1, AU 12, AU 15, AU 20, and AU 23, were calculated and analyzed using Python.This enabled the determination of the mean and standard deviation of AUs.Additionally, graphs showing the mean values and standard deviation of the AUs on the video data were generated.
The most significant presentation results reflected the level of fatigue experienced by baggage handlers.Although the analysis results continued to include the 5 AUs, they contributed to understanding the fatigue level based on the AUs observed in the videos of workers.

RESULT AND DISCUSSION
The dataset collected consisted of dynamic frontal face footage of workers.Prior to this process, workers had signed permission forms authorizing the use of their data for the research.Screenshots of sample videos were shown in Figs 4 and 5, showcasing the collections of videos featuring the baggage handlers before and after their shifts.
After successfully collecting and editing the video data for 5 seconds, the next step was to process the videos in the CERT folders.This process yielded FACS presentation data from each stimulus, producing 46 AUs and generating 5 datasets of AUs read across the 100 videos.The results of video processing before and after work in CERT were shown in Figs 6 and 7.The results of CERT were stored in the txt format, which was converted to CSV in order to remove the missing value.An overview of the data before and after cleaning was shown in Tables 1 and 2. After all video data had been cleared of missing values, the next step was to create a new table that contained the mean value of each baggage handler from the five AUs.Tables 3  and 4 showed the mean values of workers before and after work.The distribution plot of AU 01 showed that before work, the data tended to spread forming a negatively skewed distribution, but formed a positively skewed distribution after work.Similarly, Plot 12 indicated that the data tended to spread in a positively skewed manner before work, and also formed a normal curve after work.The distribution plot of AU 15 followed the same pattern, and that of AU 20 showed a positively skewed distribution both before and after work.Lastly, AU 23 indicated that the data tended to form a normal curve before work, and became positively skewed after work.An overview of the distribution plots for the five AUs could be seen below.The matplotlib library in Python was utilized for data visualization, and pandas were used to display the descriptive statistical values, including the mean and standard deviation of the five AUS (AU 01 Inner Brow Raise, AU   The AUs from FACS served as indicators of fatigue and included AU 01, AU 12, AU 15 [21], AU 20, and AU 23 [22].After applying various FACS data extraction methods, visualizations were created.These AUs played a crucial role in the formation of expressions in FACS [15].Therefore In the next step, the differences in the results of the AUs before and after working, were analyzed.The paired sample t-test was employed to determine significant differences between the conditions before and after treatment and to test for differences between the two samples.The t-test results showed a significant difference between the values of AUs, as indicated by a p-value < 0.05.The details of the t-test could be seen in Table 7. Based on the collected and analyzed video data, noticeable changes were observed in the facial expression of workers after work.The majority of subjects appeared sluggish and tired.While the recommended frequency for carrying a 50 kg load was once every 15 minutes or eight times within 2 hours [23], the reality for baggage handlers at the Yos Sudarso Port of Ambon exceeded this limit.Goods weighing an average of 50 kg were transported ten times in 2 hours.Additionally, there were downward movements around the lips and slightly upward movements of the eyebrows, indicating activity in facial muscles.Several AUs read from the videos, such as AU 1, AU 12, AU 15, AU 20, and AU 23 (mostly involving the mouth component) [24], exhibited high values that were known indicators of fatigue based on previous research [21,22].Furthermore, a difference in the value of AUs on the faces of workers before and after work was observed.This could be seen in the descriptive statistical value table of the five AUs, which revealed an increase in the mean value after working for AU 01 involving the facial muscles Pars medialis, AU 15 with facial muscles Depressor anguli oris (Triangularis), AU 20 with facial muscles Risorius, and AU 23 involving the facial muscles Orbicularis oris.It was also observed that there was a decrease in the mean value for AU 12 with facial muscles Zygomatic Major.This could be attributed to a decline in the intensity of non-Duchenne smiles among workers, which was associated with AU 12 [25].Additionally, there was an increase in the intensity of other AUs that indicated fatigue in most subjects.After comparing the average values of the five AUs indicating fatigue, tests were conducted to analyze the pre-and post-work differences in these units.This analysis included AU 01 with the Pars medialis facial muscle, AU 12 with the Zygomatic Major, AU 15 involving the Depressor anguli oris (Triangularis), AU 20 with the Risorius, AU 23 with Orbicularis oris, each yielding p-values of 0.041, 0.004, 0.000, 0.003, and 0.014, respectively.These p-values showed a significant difference between the values of five AUs, as all of them were less than 0.05.Understanding AUs that indicated fatigue could have a positive impact on ergonomics, leading to improved performance and working conditions for baggage handlers.

CONCLUSION
In conclusion, the research conducted on the fatigue level of baggage handlers using AUs showed that workers exhibited signs of fatigue.The data visualization results from the five AUs associated with fatigue, namely AU 01, AU 12, AU 15, AU 20, and AU 23, indicated an increase in intensity after work.Meanwhile, AU 12, which was associated with smiling expressions, showed a decrease in intensity among most subjects.These results showed a significant difference in the facial expressions of workers before and after work.Understanding the AUs that served as indicators of fatigue among workers could have a positive impact on all parties involved.This paper served as initial research to examine and analyze the fatigue level of workers based on the facial action coding system.Future work in this field could involve the development of an application or integrated system with a camera capable of detecting fatigue based on the output of previously identified fatigue indicator AUs.This application or system could be beneficial in enhancing the quality of work and improving the working conditions of workers.This would also aid in optimizing the performance by preventing workers from transporting goods that exceed their capacity.Despite the positive implications, this research still had some limitations, such as the duration between video recordings taken before and after work, which was limited to 2 hours.In this scenario, there was a possibility that different results could have arisen when the duration exceeded 2 hours or when workers were exposed to varying loads.
Fig 1 shows examples of AUs and their interpretations.

Fig 1 .
Fig 1.A List of AUs and their interpretations Fig 2 shows the interface of CERT.

Fig 2 .
Fig 2.CERT interface (adopted from[18]) Fig 3 showed the flowchart representing the process undertaken in this research.

Fig 3 .
Fig 3. Research flowchartThe details of each step were described below:

Fig 8 .
Fig 8. Distribution plot of all action units (AUs) (A: before working, B: after working)

Fig 9 .Fig 10 .
Fig 9. Visualization of mean (A: before working, B: after working) , mean and standard deviation visualization graphs and descriptive statistical tables showed indications of fatigue among baggage handlers.The mean values of the four fatigue-indicating AUs were higher than other AU values.AU 1, AU 12, AU 15, AU 20, and AU 23 had averages of -0.855, 0.119, 2.151, 1.535, and 1.236, respectively.Fig 9 showed an increase in the values of AU 1, AU 15, AU 20, and AU 23, while that of AU 12 decreased.

Table 1 .
Sample Dataset Worker 01 Before Cleaning

Table 3 .
Sample Mean Value of Each Worker Before Working

Table 4 .
Sample Mean Value of Each Worker After Working

Table 6 .
12 Lip Corner Pull, AU 15 Lip Corner Depressor, AU 20 Lip Stretch, AU 23 Lip Tightener).The descriptive statistical values were presented in the table below.Action Units (AUs) Descriptive

Table 7 .
T-Test Results of AUs Before and After Working