Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery

Training and assessment paradigms for laparoscopic surgical skills are evolving from traditional mentor–trainee tutorship towards structured, more objective and safer programs. Accreditation of surgeons requires reaching a consensus on metrics and tasks used to assess surgeons’ psychomotor skills. Ongoing development of tracking systems and software solutions has allowed for the expansion of novel training and assessment means in laparoscopy. The current challenge is to adapt and include these systems within training programs, and to exploit their possibilities for evaluation purposes. This paper describes the state of the art in research on measuring and assessing psychomotor laparoscopic skills. It gives an overview on tracking systems as well as on metrics and advanced statistical and machine learning techniques employed for evaluation purposes. The later ones have a potential to be used as an aid in deciding on the surgical competence level, which is an important aspect when accreditation of the surgeons in particular, and patient safety in general, are considered. The prospective of these methods and tools make them complementary means for surgical assessment of motor skills, especially in the early stages of training. Successful examples such as the Fundamentals of Laparoscopic Surgery should help drive a paradigm change to structured curricula based on objective parameters. These may improve the accreditation of new surgeons, as well as optimize their already overloaded training schedules.

Objective Assessment of Olfactory Function Using Functional Magnetic Resonance Imaging=Estudio objetivo del olfato mediante resonancia magnética funcional

Objective: To show the results of a device that generates automated olfactory stimuli suitable for functional magnetic resonance imaging (fMRI) experiments. Material and methods: Te n normal volunteers, 5 women and 5 men, were studied. The system allows the programming of several sequences, providing the capability to synchronise the onset of odour presentation with acquisition by a trigger signal of the MRI scanner. The olfactometer is a device that allows selection of the odour, the event paradigm, the time of stimuli and the odour concentration. The paradigm used during fMRI scanning consisted of 15-s blocks. The odorant event took 2 s with butanol, mint and coffee. Results: We observed olfactory activity in the olfactory bulb, entorhinal cortex (4%), amygdala (2.5%) and temporo-parietal cortex, especially in the areas related to emotional integration. Conclusions: The device has demonstrated its effectiveness in stimulating olfactory areas and its capacity to adapt to fMRI equipment.RESUMEN Objetivo: Mostrar los resultados del olfatómetro capaz de generar tareas olfativas en un equipo de resonancia magnética funcional (fMRI). Material y métodos: Estudiamos 10 sujetos normales: 5 varones y 5 mujeres. El olfatómetro está dise ̃ nado para que el estímulo que produce se sincronice con el equipo de fMRI mediante la se ̃ nal desencadenante que suministra el propio equipo. El olfatómetro es capaz de: selec- cionar el olor, secuenciar los distintos olores, programar la frecuencia y duración de los olores y controlar la intensidad del olor. El paradigma utilizado responde a un dise ̃ no de activación asociada a eventos, en el que la duración del bloque de activación y de reposo es de 15 s. La duración del estímulo olfativo (butanol, menta o café) es de 2 segundos, durante toda la serie que consta de 9 ciclos. Resultados: Se ha observado reactividad (contraste BOLD) en las diferentes áreas cerebrales involucradas en las tareas olfativas: bulbo olfatorio, córtex entorrinal (4%), amigdala (2,5%) y córtex temporoparietal. Las áreas relacionadas con integración de las emociones tienen una reactividad mayor. Conclusiones: El dispositivo propuesto nos permite controlar de forma automática y sincronizada los olores necesarios para estudiar la actividad de las áreas olfatorias cerebrales mediante fMRI.

Face, content, and construct validity of the EndoViS training system for objective assessment of psychomotor skills of laparoscopic surgeons

Background The aim of this study is to present face, content, and constructs validity of the endoscopic orthogonal video system (EndoViS) training system and determines its efficiency as a training and objective assessment tool of the surgeons’ psychomotor skills. Methods Thirty-five surgeons and medical students participated in this study: 11 medical students, 19 residents, and 5 experts. All participants performed four basic skill tasks using conventional laparoscopic instruments and EndoViS training system. Subsequently, participants filled out a questionnaire regarding the design, realism, overall functionality, and its capabilities to train hand–eye coordination and depth perception, rated on a 5-point Likert scale. Motion data of the instruments were obtained by means of two webcams built into a laparoscopic physical trainer. To identify the surgical instruments in the images, colored markers were placed in each instrument. Thirteen motion-related metrics were used to assess laparoscopic performance of the participants. Statistical analysis of performance was made between novice, intermediate, and expert groups. Internal consistency of all metrics was analyzed with Cronbach’s α test. Results Overall scores about features of the EndoViS system were positives. Participants agreed with the usefulness of tasks and the training capacities of EndoViS system (score >4). Results presented significant differences in the execution of three skill tasks performed by participants. Seven metrics showed construct validity for assessment of performance with high consistency levels. Conclusions EndoViS training system has been successfully validated. Results showed that EndoViS was able to differentiate between participants of varying laparoscopic experience. This simulator is a useful and effective tool to objectively assess laparoscopic psychomotor skills of the surgeons.

Assessment of free-living physical activity in humans: an overview of currently available and proposed new measures.

The number of physical activity measures and indexes used in the human literature is large and may result in some difficulty for the average investigator to choose the most appropriate measure. Accordingly, this review is intended to provide information on the utility and limitations of the various measures. Its primary focus is the objective assessment of free-living physical activity in humans based on physiological and biomechanical methods. The physical activity measures have been classified into three categories: Measures based on energy expenditure or oxygen uptake, such as activity energy expenditure, activity-related time equivalent, physical activity level, physical activity ratio, metabolic equivalent, and a new index of potential interest, daytime physical activity level. Measures based on heart rate monitoring, such as net heart rate, physical activity ratio heart rate, physical activity level heart rate, activity-related time equivalent, and daytime physical activity level heart rate. Measures based on whole-body accelerometry (counts/U time). Quantification of the velocity and duration of displacement in outdoor conditions by satellites using the Differential Global Positioning System may constitute a surrogate for physical activity, because walking is the primary activity of man in free-living conditions. A general outline of the measures and indexes described above is presented in tabular form, along with their respective definition, usual applications, advantages, and shortcomings. A practical example is given with typical values in obese and non-obese subjects. The various factors to be considered in the selection of physical activity methods include experimental goals, sample size, budget, cultural and social/environmental factors, physical burden for the subject, and statistical factors, such as accuracy and precision. It is concluded that no single current technique is able to quantify all aspects of physical activity under free-living conditions, requiring the use of complementary methods. In the future, physical activity sensors, which are of low-cost, small-sized, and convenient for subjects, investigators, and clinicians, are needed to reliably monitor, during extended periods in free-living situations, small changes in movements and grade as well as duration and intensity of typical physical activities.

A model for physical activity, bahaviour and energy expenditure assessment in real-life condition

RésuméL'origine de l'obésité, qui atteint des proportions épidémiques, est complexe. Elle est liée au mode de vie et au comportement des individus par rapport à l'activité physique, expression des choix individuels et de l'interaction avec l'environnement. Les mesures du comportement au niveau de l'activité physique des individus face à leur environnement, la répartition des types d'activité physique, la durée, la fréquence, l'intensité, et la dépense énergétique sont d'une grande importance. Aujourd'hui, il y a un manque de méthodes permettant une évaluation précise et objective de l'activité physique et du comportement des individus. Afin de compléter les recherches relatives à l'activité physique, à l'obésité et à certaines maladies, le premier objectif du travail de thèse était de développer un modèle pour l'identification objective des types d'activité physique dans des conditions de vie réelles et l'estimation de la dépense énergétique basée sur une combinaison de 2 accéléromètres et 1 GPS. Le modèle prend en compte qu'une activité donnée peut être accomplie de différentes façons dans la vie réelle. Les activités quotidiennes ont pu être classées en 8 catégories, de sédentaires à actives, avec une précision de 1 min. La dépense énergétique a pu peut être prédite avec précision par le modèle. Après validation du modèle, le comportement des individus de l'activité physique a été évalué dans une seconde étude. Nous avons émis l'hypothèse que, dans un environnement caractérisé par les pentes, les personnes obèses sont tentées d'éviter les pentes raides et de diminuer la vitesse de marche au cours d'une activité physique spontanée, ainsi que pendant les exercices prescrits et structurés. Nous avons donc caractérisé, par moyen du modèle développé, le comportement des individus obèses dans un environnement vallonné urbain. La façon dont on aborde un environnement valloné dans les déplacements quotidiens devrait également être considérée lors de la prescription de marche supplémentaire afin d'augmenter l'activité physique.SummaryOrigin of obesity, that reached epidemic proportion, is complex and may be linked to different lifestyle and physical activity behaviour. Measurement of physical activity behaviour of individuals towards their environment, the distribution of physical activity in terms of physical activity type, volume, duration, frequency, intensity, and energy expenditure is of great importance. Nowadays, there is a lack of methods for accurate and objective assessment of physical activity and of individuals' physical activity behaviour. In order to complement the research relating physical activity to obesity and related diseases, the first aim of the thesis work was to develop a model for objective identification of physical activity types in real-life condition and energy expenditure based on a combination of 2 accelerometers and 1 GPS device. The model takes into account that a given activity can be achieved in many different ways in real life condition. Daily activities could be classified in 8 categories, as sedentary to active physical activity, within 1 min accuracy, and physical activity patterns determined. The energy expenditure could be predicted accurately with an accuracy below 10%. Furthermore, individuals' physical activity behaviour is expression of individual choices and their interaction with the neighbourhood environment. In a second study, we hypothesized that, in an environment characterized by inclines, obese individuals are tempted to avoid steep positive slopes and to decrease walking speed during spontaneous outdoor physical activity, as well as during prescribed structured bouts of exercise. Finally, we characterized, by mean of the developed model, the physical activity behaviour of obese individuals in a hilly urban environment. Quantifying how one tackles hilly environment or avoids slope in their everyday displacements should be also considered while prescribing extra walking in free-living conditions in order to increase physical activity.

Video Quality Assessments on Digital TV and Video Streaming Services Using Objective Metrics

There is a wide range of telecommunications services that transmit voice, video and data through complex transmission networks and in some cases, the service has not an acceptable quality level for the end user. In this sense the study of methods for assessing video quality and voice have a very important role. This paper presents a classification scheme, based on different criteria, of the methods and metrics that are being studied in recent years. This paper presents how the video quality is affected by degradation in the transmission channel in two kinds of services: Digital TV (ISDB-TB) due the fading in the air interface and video streaming service on an IP network due packet loss. For Digital TV tests was set up a scenario where the digital TV transmitter is connected to an RF channel emulator, where are inserted different fading models and at the end, the videos are saved in a mobile device. The tests of streaming video were performed in an isolated scenario of IP network, which are scheduled several network conditions, resulting in different qualities of video reception. The video quality assessment is performed using objective assessment methods: PSNR, SSIM and VQM. The results show how the losses in the transmission channel affects the quality of end-user experience on both services studied.

Quantitative assessment of the intensity of palmar and plantar sweating in patients with primary palmoplantar hyperhidrosis

Objective: To compare individuals with and without hyperhidrosis in terms of the intensity of palmar and plantar sweating. Methods: We selected 50 patients clinically diagnosed with palmoplantar hyperhidrosis and 25 normal individuals as controls. We quantified sweating using a portable noninvasive electronic device that has relative humidity and temperature sensors to measure transepidermal water loss. All of the individuals had a body mass index of 20-25 kg/cm(2). Subjects remained at rest for 20-30 min before the measurements in order to reduce external interference. The measurements were carried out in a climate-controlled environment (21-24 degrees C). Measurements were carried out on the hypothenar region on both hands and on the medial plantar region on both feet. Results: In the palmoplantar hyperhidrosis group, the mean transepidermal water loss on the hands and feet was 133.6 +/- 51.0 g/m(2)/h and 71.8 +/- 40.3 g/m(2)/h, respectively, compared with 37.9 +/- 18.4 g/m(2)/h and 27.6 +/- 14.3 g/m(2)/h, respectively, in the control group. The differences between the groups were statistically significant (p < 0.001 for hands and feet). Conclusions: This method proved to be an accurate and reliable tool to quantify palmar and plantar sweating when performed by a trained and qualified professional.

Benefits of cardiac surgery in octogenarians--a postoperative quality of life assessment

OBJECTIVE: Assessment of postoperative quality of life in patients over 80 years after cardiac surgery including coronary artery bypass grafting (CABG), aortic valve replacement (AVR) and combined procedures. METHODS: Quality of life of n=136 patients over 80 years at operation (82.3+/-2.1 years), undergoing isolated CABG in 61 patients (45%), isolated AVR in 34 patients (25%) and a combination of CABG and AVR in 41 patients (30%) between January 1999 and December 2003 was reviewed. Preoperatively 66.2% presented in NYHA-class III/IV or CCS-class III/IV. Mean ejection fraction (EF) was 59.5%+/-14.0 (range 25-90%). Quality of life assessment was performed via a Seattle Angina Questionnaire. Follow-up was 100% complete for a total of 890 days (69-1853 days). RESULTS: Five-year survival was 70% for the CABG group, 75% for the AVR group and 65% for the CABG/AVR group. Quality of life was remarkable in all of the three groups after surgery. Overall 97 patients (81%) were not or little disabled in their daily activity. One hundred and twelve patients (93%) were free or considerably less symptomatic. Seventy-eight patients or 65% reported to be very satisfied with their current quality of life and 112 patients (93%) felt very reassured to have continuous full access to medical treatment despite of their advanced age. CONCLUSIONS: A remarkable quality of life and important improvement in the functional status after cardiac surgery in patients over 80 paired with a satisfactory medium-term survival justify early intervention for heart disease in this age group. Therefore, referral practice for patients over 80 years for heart surgery should be handled liberally.

Children's and adolescent's self - assessment of metabolic control versus professional judgment: a cross-sectional retrospective and prospective cohort study

BACKGROUND Morbidity and mortality in T1DM depend on metabolic control, which is assessed by HbA1c measurements every 3-4 months. Patients' self-perception of glycemic control depends on daily blood glucose monitoring. Little is known about the congruence of patients' and professionals' perception of metabolic control in T1DM. OBJECTIVE To assess the actual patients' self-perception and objective assessment (HbA1c) of metabolic control in T1DM children and adolescents and to investigate the possible factors involved in any difference. METHODS Patients with T1DM aged 8 - 18 years were recruited in a cross-sectional, retrospective and prospective cohort study. Data collection consisted of clinical details, measured HbA1c, self-monitored blood glucose values and questionnaires assessing self and professionals' judgment of metabolic control. RESULTS 91 patients participated. Mean HbA1c was 8.03%. HbA1c was higher in patients with a diabetes duration > 2 years (p = 0.025) and in patients of lower socioeconomic level (p = 0.032). No significant correlation was found for self-perception of metabolic control in well and poorly controlled patients. We found a trend towards false-positive memory of the last HbA1c in patients with a HbA1c > 8.5% (p = 0.069) but no difference in patients' knowledge on target HbA1c between well and poorly controlled patients. CONCLUSIONS T1DM patients are aware of a target HbA1c representing good metabolic control. Ill controlled patients appear to have a poorer recollection of their HbA1c. Self-perception of actual metabolic control is similar in well and poorly controlled T1DM children and adolescents. Therefore, professionals should pay special attention that ill controlled T1DM patients perceive their HbA1c correctly.

Systems and technologies for objective evaluation of technical skills in laparoscopic surgery

Minimally invasive surgery is a highly demanding surgical approach regarding technical requirements for the surgeon, who must be trained in order to perform a safe surgical intervention. Traditional surgical education in minimally invasive surgery is commonly based on subjective criteria to quantify and evaluate surgical abilities, which could be potentially unsafe for the patient. Authors, surgeons and associations are increasingly demanding the development of more objective assessment tools that can accredit surgeons as technically competent. This paper describes the state of the art in objective assessment methods of surgical skills. It gives an overview on assessment systems based on structured checklists and rating scales, surgical simulators, and instrument motion analysis. As a future work, an objective and automatic assessment method of surgical skills should be standardized as a means towards proficiency-based curricula for training in laparoscopic surgery and its certification.

Objective and psychophysical studies of infant visual development

Distortion or deprivation of vision during an early `critical' period of visual development can result in permanent visual impairment which indicates the need to identify and treat visually at-risk individuals early. A significant difficulty in this respect is that conventional, subjective methods of visual acuity determination are ineffective before approximately three years of age. In laboratory studies, infant visual function has been quantified precisely, using objective methods based on visual evoked potentials (VEP), preferential looking (PL) and optokinetic nystagmus (OKN) but clinical assessment of infant vision has presented a particular difficulty. An initial aim of this study was to evaluate the relative clinical merits of the three techniques. Clinical derivatives were devised, the OKN method proved unsuitable but the PL and VEP methods were evaluated in a pilot study. Most infants participating in the study had known ocular and/or neurological abnormalities but a few normals were included for comparison. The study suggested that the PL method was more clinically appropriate for the objective assessment of infant acuity. A study of normal visual development from birth to one year was subsequently conducted. Observations included cycloplegic refraction, ophthalmoscopy and preferential looking visual acuity assessment using horizontally and vertically oriented square wave gratings. The aims of the work were to investigate the efficiency and sensitivity of the technique and to study possible correlates of visual development. The success rate of the PL method varied with age; 87% of newborns and 98% of infants attending follow-up successfully completed at least one acuity test. Below two months monocular acuities were difficult to secure; infants were most testable around six months. The results produced were similar to published data using the acuity card procedure and slightly lower than, but comparable with acuity data derived using extended PL methods. Acuity development was not impaired in infants found to have retinal haemorrhages as newborns. A significant relationship was found between newborn binocular acuity and anisometropia but not with other refractive findings. No strong or consistent correlations between grating acuity and refraction were found for three, six or twelve months olds. Improvements in acuity and decreases in levels of hyperopia over the first week of life were suggestive of recovery from minor birth trauma. The refractive data was analysed separately to investigate the natural history of refraction in normal infants. Most newborns (80%) were hyperopic, significant astigmatism was found in 86% and significant anisometropia in 22%. No significant alteration in spherical equivalent refraction was noted between birth and three months, a significant reduction in hyperopia was evident by six months and this trend continued until one year. Observations on the astigmatic component of the refractive error revealed a rather erratic series of changes which would be worthy of further investigation since a repeat refraction study suggested difficulties in obtaining stable measurements in newborns. Astigmatism tended to decrease between birth and three months, increased significantly from three to six months and decreased significantly from six to twelve months. A constant decrease in the degree of anisometropia was evident throughout the first year. These findings have implications for the correction of infantile refractive error.

Evaluation of hemifield sector analysis protocol in multifocal visual evoked potential (MFVEP) objective perimetry for the diagnosis and early detection of glaucomatous field defects

Visual field assessment is a core component of glaucoma diagnosis and monitoring, and the Standard Automated Perimetry (SAP) test is considered up until this moment, the gold standard of visual field assessment. Although SAP is a subjective assessment and has many pitfalls, it is being constantly used in the diagnosis of visual field loss in glaucoma. Multifocal visual evoked potential (mfVEP) is a newly introduced method used for visual field assessment objectively. Several analysis protocols have been tested to identify early visual field losses in glaucoma patients using the mfVEP technique, some were successful in detection of field defects, which were comparable to the standard SAP visual field assessment, and others were not very informative and needed more adjustment and research work. In this study, we implemented a novel analysis approach and evaluated its validity and whether it could be used effectively for early detection of visual field defects in glaucoma. OBJECTIVES: The purpose of this study is to examine the effectiveness of a new analysis method in the Multi-Focal Visual Evoked Potential (mfVEP) when it is used for the objective assessment of the visual field in glaucoma patients, compared to the gold standard technique. METHODS: 3 groups were tested in this study; normal controls (38 eyes), glaucoma patients (36 eyes) and glaucoma suspect patients (38 eyes). All subjects had a two standard Humphrey visual field HFA test 24-2 and a single mfVEP test undertaken in one session. Analysis of the mfVEP results was done using the new analysis protocol; the Hemifield Sector Analysis HSA protocol. Analysis of the HFA was done using the standard grading system. RESULTS: Analysis of mfVEP results showed that there was a statistically significant difference between the 3 groups in the mean signal to noise ratio SNR (ANOVA p<0.001 with a 95% CI). The difference between superior and inferior hemispheres in all subjects were all statistically significant in the glaucoma patient group 11/11 sectors (t-test p<0.001), partially significant 5/11 (t-test p<0.01) and no statistical difference between most sectors in normal group (only 1/11 was significant) (t-test p<0.9). sensitivity and specificity of the HAS protocol in detecting glaucoma was 97% and 86% respectively, while for glaucoma suspect were 89% and 79%. DISCUSSION: The results showed that the new analysis protocol was able to confirm already existing field defects detected by standard HFA, was able to differentiate between the 3 study groups with a clear distinction between normal and patients with suspected glaucoma; however the distinction between normal and glaucoma patients was especially clear and significant. CONCLUSION: The new HSA protocol used in the mfVEP testing can be used to detect glaucomatous visual field defects in both glaucoma and glaucoma suspect patient. Using this protocol can provide information about focal visual field differences across the horizontal midline, which can be utilized to differentiate between glaucoma and normal subjects. Sensitivity and specificity of the mfVEP test showed very promising results and correlated with other anatomical changes in glaucoma field loss.

Objective analysis of contact lens fit

Purpose: To assess the validity and repeatability of objective compared to subjective contact lens fit analysis. Methods: Thirty-five subjects (aged 22.0. ±. 3.0 years) wore two different soft contact lens designs. Four lens fit variables: centration, horizontal lag, post-blink movement in up-gaze and push-up recovery speed were assessed subjectively (four observers) and objectively from slit-lamp biomicroscopy captured images and video. The analysis was repeated a week later. Results: The average of the four experienced observers was compared to objective measures, but centration, movement on blink, lag and push-up recovery speed all varied significantly between them (p <. 0.001). Horizontal lens centration was on average close to central as assessed both objectively and subjectively (p > 0.05). The 95% confidence interval of subjective repeatability was better than objective assessment (±0.128. mm versus ±0.168. mm, p = 0.417), but utilised only 78% of the objective range. Vertical centration assessed objectively showed a slight inferior decentration (0.371. ±. 0.381. mm) with good inter- and intrasession repeatability (p > 0.05). Movement-on-blink was lower estimated subjectively than measured objectively (0.269. ±. 0.179. mm versus 0.352. ±. 0.355. mm; p = 0.035), but had better repeatability (±0.124. mm versus ±0.314. mm 95% confidence interval) unless correcting for the smaller range (47%). Horizontal lag was lower estimated subjectively (0.562. ±. 0.259. mm) than measured objectively (0.708. ±. 0.374. mm, p <. 0.001), had poorer repeatability (±0.132. mm versus ±0.089. mm 95% confidence interval) and had a smaller range (63%). Subjective categorisation of push-up speed of recovery showed reasonable differentiation relative to objective measurement (p <. 0.001). Conclusions: The objective image analysis allows an accurate, reliable and repeatable assessment of soft contact lens fit characteristics, being a useful tool for research and optimisation of lens fit in clinical practice.

Comparison of subjective and objective methods to determine the retinal arterio-venous ratio using fundus photography

Purpose: To assess the inter and intra observer variability of subjective grading of the retinal arterio-venous ratio (AVR) using a visual grading and to compare the subjectively derived grades to an objective method using a semi-automated computer program. Methods: Following intraocular pressure and blood pressure measurements all subjects underwent dilated fundus photography. 86 monochromatic retinal images with the optic nerve head centred (52 healthy volunteers) were obtained using a Zeiss FF450+ fundus camera. Arterio-venous ratios (AVR), central retinal artery equivalent (CRAE) and central retinal vein equivalent (CRVE) were calculated on three separate occasions by one single observer semi-automatically using the software VesselMap (ImedosSystems, Jena, Germany). Following the automated grading, three examiners graded the AVR visually on three separate occasions in order to assess their agreement. Results: Reproducibility of the semi-automatic parameters was excellent (ICCs: 0.97 (CRAE); 0.985 (CRVE) and 0.952 (AVR)). However, visual grading of AVR showed inter grader differences as well as discrepancies between subjectively derived and objectively calculated AVR (all p < 0.000001). Conclusion: Grader education and experience leads to inter-grader differences but more importantly, subjective grading is not capable to pick up subtle differences across healthy individuals and does not represent true AVR when compared with an objective assessment method. Technology advancements mean we no longer rely on opthalmoscopic evaluation but can capture and store fundus images with retinal cameras, enabling us to measure vessel calibre more accurately compared to visual estimation; hence it should be integrated in optometric practise for improved accuracy and reliability of clinical assessments of retinal vessel calibres. © 2014 Spanish General Council of Optometry.

An Approach for a more Objective Evaluation of Practical Projects, Used in the Training Process

Well–prepared, adaptive and sustainably developing specialists are an important competitive advantage, but also one of the main challenges for businesses. One option of the education system for creation and development of staff adequate to the needs is the development of pro jects with topics from real economy ("Practical Projects"). The objective assessment is an essential driver and motivator, and is based on a system of well-chosen, well-defined and specific criteria and indicators. An approach to a more objective evaluation of practical projects is finding more objective weights of the criteria. A natural and reasonable approach is the accumulation of opinions of proven experts and subsequent bringing out the weights from the accumulated data. The preparation and conduction of a survey among recognized experts in the field of project-based learning in mathematics, informatics and information technologies is described. The processing of the data accumulated by applying AHP, allowed us to objectively determine weights of evaluation criteria and hence to achieve the desired objectiveness. ACM Computing Classification System (1998): K.3.2.