FAST CLUSTER-LEARNING WITH PRIOR PROBABILITY FROM BIG DATASET

Abstract

ASSOCIATION RULE MINING BY APRIOR METHOD HAS BEEN ONE OF THE POPULAR DATA MINING TECHNIQUES FOR DECADES, WHERE KNOWLEDGE IN THE FORM OF ITEM-ASSOCIATION RULES IS HARVESTED FROM A DATASET. THE QUALITY OF ITEM-ASSOCIATION RULES NEVERTHELESS DEPENDS ON THE CONCENTRATION OF FREQUENT ITEMS FROM THE INPUT DATASET. WHEN THE DATASET BECOMES LARGE, THE ITEMS ARE SCATTERED FAR APART. IT IS KNOWN FROM PREVIOUS LITERATURE THAT CLUSTERING HELPS PRODUCE SOME DATA GROUPS WHICH ARE CONCENTRATED WITH FREQUENT ITEMS. AMONG ALL THE DATA CLUSTERS GENERATED BY A CLUSTERING ALGORITHM, THERE MUST BE ONE OR MORE CLUSTERS WHICH CONTAIN SUITABLE AND FREQUENT ITEMS. IN TURN, THE ASSOCIATION RULES THAT ARE MINED FROM SUCH CLUSTERS WOULD BE ASSURED OF BETTER QUALITIES IN TERMS OF HIGH CONFIDENCE THAN THOSE MINED FROM THE WHOLE DATASET. HOWEVER, IT IS NOT KNOWN IN ADVANCE WHICH CLUSTER IS THE SUITABLE ONE UNTIL ALL THE CLUSTERS ARE TRIED BY ASSOCIATION RULE MINING. IT IS TIME CONSUMING IF THEY WERE TO BE TESTED BY BRUTE-FORCE. IN THIS PAPER, A STATISTICAL PROPERTY CALLED PRIOR PROBABILITY IS INVESTIGATED WITH RESPECT TO SELECTING THE BEST OUT OF MANY CLUSTERS BY A CLUSTERING ALGORITHM AS A PRE-PROCESSING STEP BEFORE ASSOCIATION RULE MINING. EXPERIMENT RESULTS INDICATE THAT THERE IS CORRELATION BETWEEN PRIOR PROBABILITY OF THE BEST CLUSTER AND THE RELATIVELY HIGH QUALITY OF ASSOCIATION RULES GENERATED FROM THAT CLUSTER. THE RESULTS ARE SIGNIFICANT AS IT IS POSSIBLE TO KNOW WHICH CLUSTER SHOULD BE BEST USED FOR ASSOCIATION RULE MINING INSTEAD OF TESTING THEM ALL OUT EXHAUSTIVELY.