Architecture of adaptive spam filtering based on machine learning algorithms

Spam is commonly defined as unsolicited email messages and the goal of spam filtering is to distinguish between spam and legitimate email messages. Much work has been done to filter spam from legitimate emails using machine learning algorithm and substantial performance has been achieved with some amount of false positive (FP) tradeoffs. In the case of spam detection FP problem is unacceptable sometimes. In this paper, an adaptive spam filtering model has been proposed based on Machine learning (ML) algorithms which will get better accuracy by reducing FP problems. This model consists of individual and combined filtering approach from existing well known ML algorithms. The proposed model considers both individual and collective output and analyzes them by an analyzer. A dynamic feature selection (DFS) technique also proposed in this paper for getting better accuracy.

Dynamic feature selection for spam filtering using support vector machine

Spam is commonly defined as unsolicited email messages and the goal of spam filtering is to differentiate spam from legitimate email. Much work have been done to filter spam from legitimate emails using machine learning algorithm and substantial performance has been achieved with some amount of false positive (FP) tradeoffs. In this paper, architecture of spam filtering has been proposed based on support vector machine (SVM,) which will get better accuracy by reducing FP problems. In this architecture an innovative technique for feature selection called dynamic feature selection (DFS) has been proposed which is enhanced the overall performance of the architecture with reduction of FP problems. The experimental result shows that the proposed technique gives better performance compare to similar existing techniques.

Email categorization using multi-stage classification technique

This paper presents an innovative email categorization using a serialized multi-stage classification ensembles technique. Many approaches are used in practice for email categorization to control the menace of spam emails in different ways. Content-based email categorization employs filtering techniques using classification algorithms to learn to predict spam e-mails given a corpus of training e-mails. This process achieves a substantial performance with some amount of FP tradeoffs. It has been studied and investigated with different classification algorithms and found that the outputs of the classifiers vary from one classifier to another with same email corpora. In this paper we have proposed a multi-stage classification technique using different popular learning algorithms with an analyser which reduces the FP (false positive) problems substantially and increases classification accuracy compared to similar existing techniques.

An innovative analyser for email classification based on grey list analysis

In this paper we propose a new technique of email classification based on grey list (GL) analysis of user emails. This technique is based on the analysis of output emails of an integrated model which uses multiple classifiers of statistical learning algorithms. The GL is a list of classifier/(s) output which is/are not considered as true positive (TP) and true negative (TN) but in the middle of them. Many works have been done to filter spam from legitimate emails using classification algorithm and substantial performance has been achieved with some amount of false positive (FP) tradeoffs. In the case of spam detection the FP problem is unacceptable, sometimes. The proposed technique will provide a list of output emails, called "grey list (GL)", to the analyser for making decisions about the status of these emails. It has been shown that the performance of our proposed technique for email classification is much better compare to existing systems, in order to reducing FP problems and accuracy.

Automated attention flags in chronic disease care planning

Objectives: To assess the value of computerised decision support in the management of chronic respiratory disease by comparing agreement between three respiratory specialists, general practitioners (care coordinators), and decision support software.
Methods: Care guidelines for two chronic obstructive pulmonary disease projects of the SA HealthPlus Coordinated Care Trial were formulated. Decision support software, Care Plan On-Line (CPOL), was created to represent the intent of these guidelines via automated attention flags to appear in patients' electronic medical records. For a random sample of 20 patients with care plans, decisions about the use of nine additional services (eg,.smoking cessation, pneumococcal vaccination) were compared between the respiratory specialists, the patients' GPs and the CPOL attention flags.
Results: Agreement among the specialists was at the lower end of moderate (intraclass correlation coefficient [ICC], 0.48; 95% CI, 0.39-0.56), with a 20% rate of contradictory decisions. Agreement with recommendations of specialists was moderate to poor for GPs (le, 0.49; 95% CI, 0.33-0.66) and moderate to good for CPOL (K, 0.72; 95% CI, 0.55-0.90). CPOL agreement with GPs was moderate to poor (le, 0.41; 95% CI, 0.24-0.58). GPs were less likely than specialists or CPOL to decide in favour of an additional service (P< 0.001). CPOL was 87% accurate as an indicator of specialist decisions. It gave a 16% false-positive rate according to specialist decisions, and flagged 61% of decisions where GPs said No and specialists said Yes.
Conclusions: Automated decision support may provide GPs with improved access to the intent of guidelines; however, further investigation is required.

Cost-effectiveness analysis of screening for lung cancer with low dose spiral CT (computed tomography) in the Australian setting

Introduction:
Low dose spiral computed tomography (CT) is a sensitive screening tool for lung cancer that is currently being evaluated in both non-randomised studies and randomised controlled trials.
Methods:
We conducted a quantitative decision analysis using a Markov model to determine whether, in the Australian setting, offering spiral CT screening for lung cancer to high risk individuals would be cost-effective compared with current practice. This exploratory analysis was undertaken predominantly from the perspective of the government as third-party funder. In the base-case analysis, the costs and health outcomes (life-years saved and quality-adjusted life years) were calculated in a hypothetical cohort of 10,000 male current smokers for two alternatives: (1) screen for lung cancer with annual CT for 5 years starting at age 60 year and treat those diagnosed with cancer or (2) no screening and treat only those who present with symptomatic cancer.
Results:
For male smokers aged 60–64 years, with an annual incidence of lung cancer of 552 per 100,000, the incremental cost-effectiveness ratio was $57,325 per life-year saved and $105,090 per QALY saved. For females aged 60–64 years with the same annual incidence of lung cancer, the cost-effectiveness ratio was $51,001 per life-year saved and $88,583 per QALY saved. The model was used to examine the relationship between efficacy in terms of the expected reduction in lung cancer mortality at 7 years and cost-effectiveness. In the base-case analysis lung cancer mortality was reduced by 27% and all cause mortality by 2.1%. Changes in the estimated proportion of stage I cancers detected by screening had the greatest impact on the efficacy of the intervention and the cost-effectiveness. The results were also sensitive to assumptions about the test performance characteristics of CT scanning, the proportion of lung cancer cases overdiagnosed by screening, intervention rates for benign disease, the discount rate, the cost of CT, the quality of life in individuals with early stage screen-detected cancer and disutility associated with false positive diagnoses. Given current knowledge and practice, even under favourable assumptions, reductions in lung cancer mortality of less than 20% are unlikely to be cost-effective, using a value of $50,000 per life-year saved as the threshold to define a “cost-effective” intervention.
Conclusion:
The most feasible scenario under which CT screening for lung cancer could be cost-effective would be if very high-risk individuals are targeted and screening is either highly effective or CT screening costs fall substantially.

Carpal tunnel syndrome: diagnostic usefulness of sonography

PURPOSE: To prospectively evaluate accuracy of sonography for diagnosis of carpal tunnel syndrome (CTS) in patients clinically suspected of having the disease in one or both hands.
MATERIALS AND METHODS: A prospective cohort of 133 patients suspected of having CTS were referred to a teaching hospital between October 2001 and June 2002 for electrodiagnostic study. One hundred twenty patients (98 women, 22 men; mean age, 49 years; range, 19–83 years) underwent sonography within 1 week after electrodiagnostic study. Radiologist was blinded to electrodiagnostic study results. Seventy-five patients had bilateral symptoms; 23 patients, right-hand symptoms; and 22 patients, left-hand symptoms (total, 195 symptomatic hands). Cross-sectional area of median nerve was measured at three levels: immediately proximal to carpal tunnel inlet, at carpal tunnel inlet, and at carpal tunnel outlet. Flexor retinaculum was used as a landmark to margins of carpal tunnel. Optimal threshold levels (determined with classification and regression tree analysis) for areas proximal to and at tunnel inlet and at tunnel outlet were used to discriminate between patients with and patients without disease. Sensitivity, specificity, and false-positive and false-negative rates were derived on the basis of final diagnosis, which was determined with clinical history and electrodiagnostic study results as reference standard.
RESULTS: For right hands, sonography had sensitivity of 94% (66 of 70); specificity, 65% (17 of 26); false-positive rate, 12% (nine of 75); and false-negative rate, 19% (four of 21) (cutoff, 0.09 cm2 proximal to tunnel inlet and 0.12 cm2 at tunnel outlet). For left hands, sensitivity was 83% (53 of 64); specificity, 73% (24 of 33); false-positive rate, 15% (nine of 62); and false-negative rate, 31% (11 of 35) (cutoff, 0.10 cm2 proximal to tunnel inlet).
CONCLUSION: Sonography is comparable to electrodiagnostic study in diagnosis of CTS and should be considered as initial test of choice for patients suspected of having CTS.

Automated identification of lung nodules

A system that can automatically detect nodules within lung images may assist expert radiologists in interpreting the abnormal patterns as nodules in 2D CT lung images. A system is presented that can automatically identify nodules of various sizes within lung images. The pattern classification method is employed to develop the proposed system. A random forest ensemble classifier is formed consisting of many weak learners that can grow decision trees. The forest selects the decision that has the most votes. The developed system consists of two random forest classifiers connected in a series fashion. A subset of CT lung images from the LIDC database is employed. It consists of 5721 images to train and test the system. There are 411 images that contained expert- radiologists identified nodules. Training sets consisting of nodule, non-nodule, and false-detection patterns are constructed. A collection of test images are also built. The first classifier is developed to detect all nodules. The second classifier is developed to eliminate the false detections produced by the first classifier. According to the experimental results, a true positive rate of 100%, and false positive rate of 1.4 per lung image are achieved.

A random forest for lung nodule identification

A method is presented for identification of lung nodules. It includes three stages: image acquisition, background removal, and nodule detection. The first stage improves image quality. The second stage extracts long lobe regions. The third stage detects lung nodules. The method is based on the random forest learner. Training set contains nodule, non-nodule, and false-positive patterns. Test set contains randomly selected images. The developed method is compared against the support vector machine. True-positives of 100% and 85.9%, and false-positives of 1.27 and 1.33 per image were achieved by the developed method and the support vector machine, respectively.

Can the ABCD Score be dichotomised to identify high−risk patients with trainsient ischaemic attack in the emergency department?

Background: Recent evidence shows a substantial short-term risk of ischaemic stroke after transient ischaemic attack (TIA). Identification of patients with TIA with a high short-term risk of stroke is now possible through the use of the “ABCD Score”, which considers age, blood pressure, clinical features and duration of symptoms predictive of stroke.

Aim: To evaluate the ability of dichotomising the ABCD Score to predict stroke at 7 and 90 days in a population with TIA presenting to an emergency department.

Methods: A retrospective audit was conducted on all probable or definite TIAs presenting to the emergency department of a metropolitan hospital from July to December 2004. The ABCD Score was applied to 98 consecutive patients with TIA who were reviewed for subsequent strokes within 90 days. Patients obtaining an ABCD Score ≥5 were considered to be at high risk for stroke.

Results: Dichotomising the ABCD Score categorised 48 (49%) patients with TIA at high risk for stroke (ABCD Score ≥5). This high-risk group contained all four strokes that occurred within 7 days (sensitivity 100% (95% confidence interval (CI) 40% to 100%), specificity 53% (95% CI 43% to 63%), positive predictive value 8% (95% CI 3% to 21%) and negative predictive value 100% (95% CI 91% to 100%)), and six of seven occurring within 90 days (sensitivity 86% (95% CI 42% to 99%), specificity 54% (95% CI 43% to 64%), positive predictive value 12.5% (95% CI 5% to 26%) and negative predictive value 98% (95% CI 88% to 100%)). Removal of the “age” item from the ABCD Score halved the number of false-positive cases without changing its predictive value for stroke.

Conclusion: In this retrospective analysis, dichotomising the ABCD Score was overinclusive but highly predictive in identifying patients with TIA at a high short-term risk of stroke. Use of the ABCD Score in the emergency care of patients with TIA is simple, efficient and provides a unique opportunity to prevent stroke in this population of patients.

Multi-classifier classification of spam email on an ubiquitous multi-core architecture

This paper presents an innovative fusion based multi-classifier email classification on a ubiquitous multi-core architecture. Many approaches use text-based single classifiers or multiple weakly trained classifiers to identify spam messages from a large email corpus. We build upon our previous work on multi-core by apply our ubiquitous multi-core framework to run our fusion based multi-classifier architecture. By running each classifier process in parallel within their dedicated core, we greatly improve the performance of our proposed multi-classifier based filtering system. Our proposed architecture also provides a safeguard of user mailbox from different malicious attacks. Our experimental results show that we achieved an average of 30% speedup at the average cost of 1.4 ms. We also reduced the instance of false positive, which is one of the key challenges in spam filtering system, and increases email classification accuracy substantially compared with single classification techniques.

Email categorization using (2+1)-tier classification algorithms

In this paper we have proposed a spam filtering technique using (2+1)-tier classification approach. The main focus of this paper is to reduce the false positive (FP) rate which is considered as an important research issue in spam filtering. In our approach, firstly the email message will classify using first two tier classifiers and the outputs will appear to the analyzer. The analyzer will check the labeling of the output emails and send to the corresponding mailboxes based on labeling, for the case of identical prediction. If there are any misclassifications occurred by first two tier classifiers then tier-3 classifier will invoked by the analyzer and the tier-3 will take final decision. This technique reduced the analyzing complexity of our previous work. It has also been shown that the proposed technique gives better performance in terms of reducing false positive as well as better accuracy.

Spam filtering using classification algorithms

This thesis proposes an innovative adaptive multi-classifier spam filtering model, with a grey-list analyser and a dynamic feature selection method, to overcome false-positive problems in email classification. It also presents additional techniques to minimize the added complexity. Empirical evidence indicates the success of this model over existing approaches.

An abnormal-based approach to effectively detect DDOS attacks

Distributed Denial-of-Service (DDoS) attacks are a serious threat to the safety and security of cyberspace. In this paper we propose a novel metric to detect DDoS attacks in the Internet. More precisely, we use the function of order α of the generalized (Rényi) entropy to distinguish DDoS attacks traffic from legitimate network traffic effectively. In information theory, entropies make up the basis for distance and divergence measures among various probability densities. We design our abnormal-based detection metric using the generalized entropy. The experimental results show that our proposed approach can not only detect DDoS attacks early (it can detect attacks one hop earlier than using the Shannon metric while order  α =2, and two hops earlier than the Shannon metric while order α =10.) but can also reduce both the false positive rate and the false negative rate, compared with the traditional Shannon entropy metric approach.