hF :dZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlm Z m!Z!dZ"edd\Z#Z$ee#e$d\Z#Z$e%e#Z#gdZ&dhdej'DzZ(ej)*dedZ+dZ,ej)*dej-ej.gdZ/dZ0ej)*de&ej)*ded Z1d!Z2ej)*d"d#d$Z3d%Z4d&Z5d'Z6d(Z7d)Z8ej)*d*d+d,gd-Z9d.Z:ej)*d/e&d0Z;d1Zd5ej?gej?d5ggfdd4id5d6gd6d5ggfid5d6gdd7ggfgd8Z@d9ZAd:ZBd;ZCd<ZDej)*d=d>d?gej)*d@ddAgdBZEdCZFdS)DzF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)sparsestats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal ) n_samples random_state)r)kdtreeballtreebruteautoc$h|] \}}|dS)label).0_outs j/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/sklearn/cluster/tests/test_hdbscan.py r'%s KKKvq#c'lKKK outlier_typectjtjd|}ddd|}t|d}t|d}t}|dg|d<||g|d<t |}|j|k \}t|ddg||j | \}t|ddgttddttd d z} t || } t| j|j| d S) O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingc||kSNr"xys r&z#test_outlier_data..2s ar(c*tj|Sr/)npisnanr0s r&r3z#test_outlier_data..3s r(r!probrrN)r5infnanr Xcopyrfitlabels_nonzerorprobabilities_listrange) r)outlier prob_checkr!r7 X_outliermodelmissing_labels_idxmissing_probs_idx clean_indices clean_models r&test_outlier_datarM(ss F6G ('++J l +G 4E \ *6 2DIQ>)Aq6222&Ju';TBBKKMM(1a&111q!%%U1c]](;(;;M))-- - 899K{*EM-,HIIIIIr(ctt}|}tdd|}t ||t t|tz }|tksJtt|}|dksJd}tj t|5tddtdddn #1swxYwYd}d |d <d |d <tj t|5td |ddddS#1swxYwYdS)zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedT)metricr>\(\?z*The precomputed distance matrix.*has shapematchNz'The precomputed distance matrix.*valuesr)rr8r8)r8rrP)rr=r>r fit_predictrlenset OUTLIER_SETn_clusters_truer r2pytestraises ValueError)D D_originallabels n_clustersscoremsgs r&test_hdbscan_distance_matrixrcHs AAJ M 5 5 5 A A! D DFAz"""S[[;.//J  ( ( ( ( "!V , ,E D==== 7C z - - -@@}4000<>DD0$E!!E%(E%sparse_constructorctjtjt}|t j|z}t j|d}d|||k<||}| td |}tt|tz }|tksJdS)zA Tests that HDBSCAN works with sparse distance matrices. 2rOrTN)r squareformpdistr=r5maxrscoreatpercentileflatteneliminate_zerosrrUrVrWrXrY)rdr] thresholdr_r`s r&#test_hdbscan_sparse_distance_matrixrofs HN1--..ANA' R88IAa9n1A M * * * 6 6q 9 9FS[[;.//J  ( ( ( ( ( (r(ctt}tt |t z }|t ksJtt|}|dksJdS)z Tests that HDBSCAN works with feature array, including an arbitrary goodness of fit check. Note that the check is a simple heuristic. rQN) rrUr=rVrWrXrYr r2)r_r`ras r&test_hdbscan_feature_arrayrqysf YY " "1 % %FS[[;.//J  ( ( ( ( "!V , ,E D======r(algorPct|t}tt |t z }|t ksJ|dvrdSttd}dtj tj didtj tj diddidtj tj dd d  |d}t||| }|||jvrNtjt"5|tddddS#1swxYwYdS|d krNtjt(5|tddddS#1swxYwYdS|tdS) z Tests that HDBSCAN works with the expected combinations of algorithms and metrics, or raises the expected errors. ) algorithm)rrN)rrVr8p)rvw) mahalanobis seuclidean minkowski wminkowskirtrP metric_paramsr|)rrUr=rVrWrXrYrrr5eyeshapeonesget valid_metricsrZr[r\r?warns FutureWarning)rrrPr_r` ALGOS_TREESr~hdbs r&test_hdbscan_algorithmsrs@t $ $ $ 0 0 3 3FS[[;.//J  ( ( ( (    K RVAGAJ//0BGAGAJ//01XBGAGAJ$7$788   c&$  #   C [&444 ]: & &   GGAJJJ                   <   \- ( (   GGAJJJ                    s$3EE"EF//F36F3ctt}|d}t t |t z }|tksJdS)z Tests that HDBSCAN can generate a sufficiently accurate dbscan clustering. This test is more of a sanity check than a rigorous evaluation. TODO: Improve and strengthen this test if at all possible. 333333?N)rr?r=dbscan_clusteringrVrWrXrY) clustererr_r`s r&test_dbscan_clusteringrs[ a  I  ( ( - -FS[[;.//J  ( ( ( ( ( (r( cut_distance)皙??r8c0tdd}tdd}t}tjdg|d<dtjg|d<tjtjg|d<t |}||}tj ||k}t|ddgtj ||k}t|dgtttd t||zz }t ||} | |} t| ||d S) r+r-r!r,r8rrwr9)rrN)r r=r>r5r;r<rr?r flatnonzerorrCrWrD) r missing_labelinfinite_labelrGrHr_rIinfinite_labels_idx clean_idxrL clean_labelss r&#test_dbscan_clustering_outlier_datarsT &i09M&z27;NIFA;IaLrv;IaLFBF#IaL IIMM) $ $E  $ $, $ ? ?F-(?@@)Aq6222.>)ABB*QC000Ss__s+=@S+S'T'TTUUI))-- ) 455K00l0KKL|VI%677777r(c \tddd\}}t|}tdddt j|jdi |}tt|tz }|tksJd S) zQ Tests that HDBSCAN using `BallTree` works with higher-dimensional data. rfr@)rr n_featuresrrzrur8r}N) r r fit_transformrr5rrrUrVrWrXrY)Hr2r_r`s r&test_hdbscan_high_dimensionalrs r B B BDAq&&q))A BGAGAJ//0k!nn  S[[;.//J  ( ( ( ( ( (r(ctddtjtjdit}t t|tz }|tksJdS)z4 Tests that HDBSCAN using `BallTree` works. rzrur8)rPr~N) rr5rr=rrUrVrWrXrYr_r`s r&!test_hdbscan_best_balltree_metricrsqC1D1D+Ek!nn S[[;.//J  ( ( ( ( ( (r(ctttdz t}tt |t z }|dksJdS)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. r8min_cluster_sizerN)rrVr=rUrWrXrs r&test_hdbscan_no_clustersrsU c!ffqj 1 1 1 = =a @ @FS[[;.//J ??????r(c8tdttdD]u}t|t}d|D}t|dkr,t jt j||ksJvdS)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points rwr8rcg|] }|dk| S)rr")r#r!s r& z1test_hdbscan_min_cluster_size..s@@@ERKKuKKKr(rN)rDrVr=rrUr5minbincount)rr_ true_labelss r&test_hdbscan_min_cluster_sizers "!SVVQ//HH*:;;;GGJJ@@&@@@ {  q 6"+k22337GGGGG HHr(ctj}t|t}t t |tz }|tksJdS)zA Tests that HDBSCAN works when passed a callable metric. rTN) r euclideanrrUr=rVrWrXrY)rPr_r`s r&test_hdbscan_callable_metricr sY F F # # # / / 2 2FS[[;.//J  ( ( ( ( ( (r(treekdballctdd|d}tjt5|t ddddS#1swxYwYdS)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rOprims_rrPrtN)rrZr[r\r?r=)rrs r&"test_hdbscan_precomputed_non_bruters 2E42E2E2E F F FC z " "  sAAAc.ttj}t t |t z }|dksJtjt}| }t|j}t||tj dftj dffD]\}}t }||d<t|j}t t |t z }|dksJ|dt|dksJ| }||d<t|j}t||d}tjt"|5td d |d d d d S#1swxYwYd S) z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. rr,r-rrrr!z4Sparse data matrices only support algorithm `brute`.rRrrrN)rr?r=r@rVrWrXr csr_matrixr>rr5r;r<r rZr[r\) dense_labelsr` _X_sparseX_sparse sparse_labels outlier_valr)X_denserbs r&test_hdbscan_sparsers99==##+LS&&455J ????!!$$I~~HIIMM(++3M|]333(*vz&:RVY>##$ h//7 <7777 @C z - - -HH{j999==hGGGHHHHHHHHHHHHHHHHHHs%H  HHrtcddg}tdd|d\}}td|}t||j|jD],\}}}t ||d d t ||d d -t|dtjd t}|jjddksJ|jjddksJd S)zj Tests that HDBSCAN centers are calculated and stored properly, and are accurate to the data. )rgrg)@rirr)rrcenters cluster_stdboth) store_centersr8g?)rtolatol)rtrrN) r rr?zip centroids_medoids_rr=r)rtrrr$rcentercentroidmedoids r&test_hdbscan_centersrCs  :&G 1gSV W W WDAq  ' ' ' + +A . .C$'$N$N;; &qt<<<<QT::::: 6AGAJ    c!ff >  "a ' ' ' ' < a A % % % % % %r(c tjd}|dd}t dddd|}tj|d \}}t|dksJ||d kd ksJt dd ddd |}tj|d \}}t|dksJ||d kdksJdS)zS Tests that HDBSCAN single-cluster selection with epsilon works correctly. rrwr9rgeomT)rcluster_selection_epsiloncluster_selection_methodallow_single_cluster) return_countsrg ףp= ?r)rrrrrtN)r5random RandomStaterandrrUuniquerV)rng no_structurer_ unique_labelscountss r&.test_hdbscan_allow_single_cluster_with_epsilonrYs; )   " "C88C##L "%!&!  k,  IfDAAAM6 }   " " " " -2% & + + + +"&!&!  k,  IfDAAAM6 }   " " " " -2% &! + + + + + +r(cddgddgddgddgg}td|gdd\}}t|}tt |jt d |jvz }|d ksJd S) z Validate that HDBSCAN can properly cluster this difficult synthetic dataset. Note that DBSCAN fails on this (see HDBSCAN plotting example) g333333g333333?ri)皙?gffffff?皙?rr)rrrrrN)r rr?rVrWr@int)rr=r$rr`s r&test_hdbscan_better_than_dbscanrzs u~t}q!fq"g>G +++    DAq ))--  CS%%&&R3;->)?)??J ??????r(z kwargs, XrOr8rwrcHtdddi||dS)zo Tests that HDBSCAN works correctly for array-likes and precomputed inputs with non-finite points. min_samplesr8Nr")rr?)r=kwargss r&test_hdbscan_usable_inputsrs1 $$$V$$((+++++r(ctjtjd}d}t jt |5td|ddddS#1swxYwYdS)zd Tests that HDBSCAN raises the correct error when there are too few non-zero distances. )rrz#There exists points with fewer thanrRrOrTN) rrr5zerosrZr[r\rr?)r=rbs r&-test_hdbscan_sparse_distances_too_few_nonzerors "(8,,--A /C z - - ---}%%%))!,,,------------------s$A55A9<A9cd}d}tjt|5td|t dddn #1swxYwYtjt|5td|t dddn #1swxYwYt ttj ttj z }t|dkretjt|5td|dt ddddS#1swxYwYdSdS) zR Tests that HDBSCAN correctly raises an error for invalid metric choices. c|Sr/r")r1s r&r3z2test_hdbscan_tree_invalid_metric..sr(zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.rRr)rtrPNrr) rZr[r\rr?r=rCrWrrrrV)metric_callablerbmetrics_not_kds r& test_hdbscan_tree_invalid_metricrsG"kO  z - - -CC(?;;;??BBBCCCCCCCCCCCCCCC z - - -EE*_===AA!DDDEEEEEEEEEEEEEEE #h455FQ ]:S 1 1 1 I I h~a/@ A A A E Ea H H H I I I I I I I I I I I I I I I I I Is5*AAA=*B33B7:B7,0E))E-0E-ctttdz}d}tjt |5|tddddS#1swxYwYdS)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. r8)rz min_samples (.*) must be at mostrRN)rrVr=rZr[r\r?)rrbs r&!test_hdbscan_too_many_min_samplesrs c!ffqj ) ) )C -C z - - -  sA++A/2A/c t}tj|d<d}t d}t jt|5||ddddS#1swxYwYdS)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserOrTrRN) r=r>r5r<rrZr[r\r?)X_nanrbrs r&"test_hdbscan_precomputed_dense_nanrs FFHHE&E$K 4C  ' ' 'C z - - - sA99A=A=rTFepsilonrc0 d}t||ddgddgddgg\} t|}t|j|j}|dz|dz|dzh}|dzd|dzd |dzdi}t |||||  fd tt D fd tt D} tj | j  } t | d S)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrrrwrrr8condensed_treeclusterscluster_label_maprrcZi|]'}|tj|kdd(S)r)r5where)r#_yr2s r& z+test_labelling_distinct..s3KKKBHQ"W--a03KKKr(c.i|]}||Sr"r")r#rfirst_with_labelr_s r&rz+test_labelling_distinct..s&KKK2v.r23KKKr(N) r rr?r_single_linkage_tree_rr rCrWr5 vectorizerr)global_random_seedrrrr=estrrr y_to_labelsaligned_targetrr_r2s @@@r&test_labelling_distinctr sZ I 'F G G   DAq ))--  C# !C4HNA y1}i!m>S2.. . . . . %$aQ:!"# Fw'*4I y>>S2.. . . . . . .r()G__doc__numpyr5rZscipyrr scipy.spatialrsklearn.clusterrsklearn.cluster._hdbscan._treerrr sklearn.cluster._hdbscan.hdbscanr sklearn.datasetsr sklearn.metricsr sklearn.metrics.pairwiser rsklearn.neighborsrrsklearn.preprocessingr sklearn.utilsrsklearn.utils._testingrrrYr=r2r ALGORITHMSitemsrXmark parametrizerMrcr csc_matrixrorqrrrrrrrrrrrrrrr;rrrrrr rr"r(r&r(s """"""###### ?>>>>>''''''111111HHHHHHHH........000000!!!!!!FFFFFFFFzCb1111wq!!$$$1N""1%% dKK1H1B1H1J1JKKKK ):;;JJ<;J>555<-0A6CT/UVV))WV)$   ,,>22%%32-,%P ) ) )77888784 ) ) )))) H H H)))$0010!H!H!HHj11&&21&*,,,B$ M "HBHq"&kBFA;-G$H$HI M "aVaV$45 q!fq!f ,,, - - -III0   /$??QH--!/!/.-@?!/H&/&/&/&/&/r(