Fig.1 Representative cervical cytology images showing multiple cell types highlighted by annotation boxes from Liang's dataset[15]. The two images in the left column show abnormal cells in the categories of ascus, lsil, hsil and scc, and those in the right column show candida and trichomonas infections that cause the surrounding normal cells to have a similar appearance to atypical squamous cells.

Fig.2 Object samples of entities in the datasets.

Fig.3 Overall workflow of this study.

Fig.4 Architecture of the proposed Trans-YOLOv5 model. A: The details of the network structure. B: Structure of C3 and CBT3. The detailed structure of the MHSA is shown within the purple dotted box on the right side, where Rh, Rw denote the relative positional encoding of the height and the width, respectively, WQ, WK, WV are the weights of the query, the key, and the weights, respectively; + and × represent element-by-element summation and matrix multiplication, respectively. C: Structure of the original head of YOLOv5 and ADH. The detailed structure of the CBAM is shown in the red dotted box.

Fig.5 Comparison of the yolov5l model and Trans-YOLOv5 model for abnormal cell detection in cervical cytology images. Arrows highlight the areas where misdiagnosis or omission occurs with the original YOLOv5 model. The images in the first column show that the baseline model misdiagnosed scc as lsil, which is correctly identified by the proposed Trans-YOLOv5 model. The images in the second, third and fourth columns show that the Trans-YOLOv5 model model correctly diagnosed the abnormal cells that were missed by the baseline model (The images are cropped for convenience of presentation).

Fig.6 Visualization of model feature maps before and after addition of CBT3 to YOLOv5. A: Original image. B: Annotated image with category labels. C: Feature map of the original YOLOv5. D: Feature map of YOLOv5 after adding CBT3.