This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. _Goal:_ Vascular surgical procedures are challenging and require proficient suturing skills. To develop these skills, medical training simulators with objective feedback for formative assessment are gaining popularity. As hardware advancements offer more complex, unique sensors, determining effective task performance measures becomes imperative for efficient suturing training. _Methods:_ 97 subjects of varying clinical expertise completed four trials on a suturing skills measurement and feedback platform (SutureCoach). Instrument handling metrics were calculated from electromagnetic motion trackers affixed to the needle driver. _Results:_ The results of the study showed that all metrics significantly differentiated between novices (no medical experience) from both experts (attending surgeons/fellows) and intermediates (residents). Rotational motion metrics were more consistent in differentiating experts and intermediates over traditionally used tooltip motion metrics. _Conclusions:_ Our work emphasizes the importance of tool motion metrics for open suturing skills assessment and establishes groundwork to explore rotational motion for quantifying a critical facet of surgical performance. _Impact Statement_–This study aims to determine the effectiveness of metrics derived from needle driver rotational and tooltip motion tracking to determine differences in clinical expertise in open needle driving.

This article has been accepted for publication at IEEE EMBC. © 2024 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.   The efficient execution of surgical operations plays a crucial role in optimizing patient outcomes, evidencing the need for effective training methods to improve surgical skills. Medical training simulators, praised for objective, automated skill assessment, require instrumented sensors and relevant metrics for targeted feedback on important aspects of a surgical procedure. Traditional metrics that capture a single instance of force, such as peak or range, lack the characterization of the entire force profile and lose subtleties that may limit accurate evaluation of the skilled application of force, a valuable aspect of assessment in surgery. This study introduces novel force metrics inspired by motion smoothness-based measures, analyzed on an extensive dataset of 97 subjects suturing on an open vascular suturing simulator. We validated the effectiveness of these metrics by comparing the metric scores for subjects with different skill levels. Our findings highlight the value of these advanced force metrics as robust indicators of suturing performance, demonstrating their valuable potential for more accurate and objective skill assessment in surgical training.