昺h9:dZddlmZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZmZddlmZddlmZdd lmZdd lmZejeZd gZeGd d ZddhZhdZddZGddeZ eGddeZ!dS)zLoads YouTube transcript.) annotationsN)Path)AnyDictListOptionalSequenceUnion)parse_qsurlparse)Document)root_validator) dataclass) BaseLoaderz0https://www.googleapis.com/auth/youtube.readonlyceZdZUdZejdz dz Zded<ejdz dz Zded<ejdz dz Z ded<dd Z e ddZ ddZ dS)GoogleApiClientaBGeneric Google API Client. To use, you should have the ``google_auth_oauthlib,youtube_transcript_api,google`` python package installed. As the google api expects credentials you need to set up a google account and register your Service. "https://developers.google.com/docs/api/quickstart/python" Example: .. code-block:: python from langchain_community.document_loaders import GoogleApiClient google_api_client = GoogleApiClient( service_account_path=Path("path_to_your_sec_file.json") ) z .credentialszcredentials.jsonrcredentials_pathservice_account_pathz token.json token_pathreturnNonec8||_dSN)_load_credentialscredsselfs t/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/langchain_community/document_loaders/youtube.py __post_init__zGoogleApiClient.__post_init__-s++-- valuesDict[str, Any]cx|ds$|dstd|S)DValidate that either folder_id or document_ids is set, but not both.rr-Must specify either channel_name or video_idsget ValueErrorclsr!s r#validate_channel_or_videoIds_is_setz3GoogleApiClient.validate_channel_or_videoIds_is_set0sI zz,-- Nfjj "7 7  NLMM M r rc6 ddlm}ddlm}ddlm}ddlm}ddlm }n#t$rtdwxYwd}|j r,|j t|j S|j r-|t|jt"}|r|js|r,|jr%|jr||nC|t|jt"}|d }t3|jd 5}||dddn #1swxYwY|S) zLoad credentials.r)Request)service_account) Credentials)InstalledAppFlowYouTubeTranscriptApiYou must run`pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib youtube-transcript-api` to use the Google Drive loaderN)portw)google.auth.transport.requestsr- google.oauth2r.google.oauth2.credentialsr/google_auth_oauthlib.flowr0youtube_transcript_apir2 ImportErrorrexistsfrom_service_account_filestrrfrom_authorized_user_fileSCOPESvalidexpired refresh_tokenrefreshfrom_client_secrets_filerrun_local_serveropenwriteto_json) rr-r.r/r0r2rflowtokens rrz!GoogleApiClient._load_credentials<s  > > > > > > 5 5 5 5 5 5 = = = = = = B B B B B B C C C C C C C   1    $ + + - - ".HHD-..  ? ! ! # # X99#do:N:NPVWWE -EK - 6 65+> 6 ggii(((('@@-..--1-55dos++ -u EMMOO,,, - - - - - - - - - - - - - - - s!;(FFFNrrr!r"rr")rr)__name__ __module__ __qualname____doc__rhomer__annotations__rrrrr+rr rrrs&'TY[[>9vid.plusyoutu.be youtube.com m.youtube.comwww.youtube.comwww.youtube-nocookie.comurlr>r Optional[str]ct|}|jtvrdS|jtvrdS|j}|drD|j}t|}d|vr(|d}t|tr|n|d}n7dS|j d}| dd}t|dkrdS|S)zEParse a youtube url and return the video id if valid, otherwise None.Nz/watchvr/ )r schemeALLOWED_SCHEMASnetlocALLOWED_NETLOCKpathendswithqueryr isinstancer>lstripsplitlen)r] parsed_urlrhrj parsed_queryidsvideo_ids r_parse_video_idrsps#J//t//t ?D }}X '  ,  s#C(c22>ssAHH4%%c**::c??2& 8}}t Or cdeZdZdZ dddZeddZeddZddZ ddZ dS) YoutubeLoaderzLoad `YouTube` transcripts.FenNrrr>add_video_infoboollanguageUnion[str, Sequence[str]] translationr^continue_on_failurec||_||_||_t|tr |g|_n||_||_||_dS)z!Initialize with YouTube video ID.N)rrrwryrkr>r{r|)rrrrwryr{r|s r__init__zYoutubeLoader.__init__sT! ,  h $ $ %%JDMM$DM&#6   r youtube_urlrcLt|}|std||S)z%Extract video id from common YT urls.z-Could not determine the video ID for the URL )rsr()rrrs rextract_video_idzYoutubeLoader.extract_video_ids=#;// M MM r kwargsrc >||}||fi|S)zGiven youtube URL, load video.)r)r*rrrrs rfrom_youtube_urlzYoutubeLoader.from_youtube_urls/'' 44s8&&v&&&r List[Document]c^ ddlm}m}m}n#t$rt dwxYwd|ji}|jr)|}|| | |j}n #|$rgcYSwxYw | |j }n!#|$r| dg}YnwxYw|j | |j }|}dd|D}t!|| gS) Load documents.r)NoTranscriptFoundTranscriptsDisabledr2ztCould not import youtube_transcript_api python package. Please install it with `pip install youtube-transcript-api`.sourcervN cDg|]}|ddStextrstrip.0ts r z&YoutubeLoader.load..s(OOOqys33OOOr  page_contentmetadata)r:rrr2r;rrrw_get_video_infoupdatelist_transcriptsfind_transcriptryr{ translatefetchjoinr ) rrrr2r video_infotranscript_list transcripttranscript_piecess rloadzYoutubeLoader.loads                O   dm,   (--//J OOJ ' ' ' 2CCDMRROO"   III  A(88GGJJ  A A A(88$@@JJJ A   '#--d.>??J&,,..XXOO=NOOOPP j8DDDEEs, '$A??B B  B((CCdictc  ddlm}n#t$rtdwxYw|d|j}|jpd|jpd|jpd|jpd|jr|j dnd|j pd|j pdd}|S)zGet important video information. Components are: - title - description - thumbnail url, - publish_date - channel_author - and more. r)YouTubezTCould not import pytube python package. Please install it with `pip install pytube`.z https://www.youtube.com/watch?v=Unknownz%Y-%m-%d %H:%M:%S)title description view_count thumbnail_url publish_datelengthauthor) pytuberr;rrrrviewsrrstrftimerr)rrytrs rrzYoutubeLoader._get_video_infos  & & & & & & &   ?   WG GG H HX*>6Y(-a-:BO445HIIIin1i,9   s #)FrvNF) rrr>rwrxryrzr{r^r|rx)rr>rr>)rr>rrrrurr)rr) rNrOrPrQr~ staticmethodr classmethodrrrrTr rrurus%% %.2%)$) 77777&\'''[' 'F'F'F'FRr ruceZdZUdZded<dZded<dZded<d Zd ed <d Zd ed<dZ d ed<d$dZ d%dZ e d&dZ d'dZd(dZd)dZd*d"Zd+d#ZdS),GoogleApiYoutubeLoaderaLoad all Videos from a `YouTube` Channel. To use, you should have the ``googleapiclient,youtube_transcript_api`` python package installed. As the service needs a google_api_client, you first have to initialize the GoogleApiClient. Additionally you have to either provide a channel name or a list of videoids "https://developers.google.com/docs/api/quickstart/python" Example: .. code-block:: python from langchain_community.document_loaders import GoogleApiClient from langchain_community.document_loaders import GoogleApiYoutubeLoader google_api_client = GoogleApiClient( service_account_path=Path("path_to_your_sec_file.json") ) loader = GoogleApiYoutubeLoader( google_api_client=google_api_client, channel_name = "CodeAesthetic" ) load.load() rgoogle_api_clientNr^ channel_namezOptional[List[str]] video_idsTrxrwrvr>captions_languageFr|rrcN||jj|_dSr)_build_youtube_clientrryoutube_clientrs rrz$GoogleApiYoutubeLoader.__post_init__$s$"889O9UVVr rrct ddlm}ddlm}n#t$rt dwxYw|dd|S)Nr)buildr1r3youtubev3) credentials)googleapiclient.discoveryrr:r2r;)rrrr2s rrz,GoogleApiYoutubeLoader._build_youtube_client'sy  7 7 7 7 7 7 C C C C C C C   1  uY%8888s )r!r"cx|ds$|dstd|S)r$rrr%r&r)s rr+z:GoogleApiYoutubeLoader.validate_channel_or_videoIds_is_set7sA zz.)) N&**[2I2I NLMM M r rrc0ddlm}m}||} ||jg}n*#|$r"|D]}||j}YnwxYw|}dd|DS)Nr)rr2rcDg|]}|ddSrrrs rrzGGoogleApiYoutubeLoader._get_transcripe_for_video_id..Ls(III!6--IIIr ) r:rr2rrrrrr)rrrrr2rravailable_transcriptrs r_get_transcripe_for_video_idz3GoogleApiYoutubeLoader._get_transcripe_for_video_id@sRRRRRRRR.??II (88$:P9QRRJJ    (7  $1;;D>   & & ( ( T!WYY !#''003    r c|jd|dd}|}|dddd}|S)Nrchannel)rqtype maxResultsrr channelId)rsearchrr)rrrequestresponse channel_ids r_get_channel_idz&GoogleApiYoutubeLoader._get_channel_id]sg%,,..33 4   ??$$g&q)$/ < r rrc | ddlm}m}n#t$rtdwxYw||}|jd|d}g}|K|}|dD]} | d d sd | dd i} |j r6| d  d | | d  | | dd } |t| | #||f$rC} |jr0t"d d| dd d| zn| Yd} ~ d} ~ wwxYw|j||}|K|S)Nr)rrzTYou must run`pip install --upgrade youtube-transcript-api` to use the youtube loaderr2)rrrrrvideoIdsnippet thumbnailsrzError fetching transscript rz , exception: )r:rrr;rrrrrr'rwpoprrappendr r|loggererror list_next) rrrrrrrrritem meta_datares r_get_document_for_channelz0GoogleApiYoutubeLoader._get_document_for_channelhsb             ,  ))'22 %,,..33 4    !((H!)  Dz~~i00&T 9(=> &6O'' 555$$T)_555#'#D#DT 9-$$L$$ )5%. ,->?/  9I$t*Y"7IIaIIJ  DDDD)0022<.s7 33H==r r%)rextendrrr()r document_lists` rrzGoogleApiYoutubeLoader.loads   N  !?!?@Q!R!R S S S S ^ N  $(N    LMM Mr rL)rrrrrM)rrr>rr>)rrr>rrrr )rr>rr>)rr>rrrrr)rNrOrPrQrSrrrwrr|rrrr+rrrrrrTr rrrs:8'&&&"&L&&&&%)I))))N!!!!! %%%%%WWWW9999 ^ K K K K         5555nr r)r]r>rr^)"rQ __future__rloggingpathlibrtypingrrrrr r urllib.parser r langchain_core.documentsr langchain_core.pydantic_v1r&langchain_core.pydantic_v1.dataclassesr)langchain_community.document_loaders.baser getLoggerrNrr@rrergrsrurrTr rrs""""""================++++++++------555555<<<<<<@@@@@@  8 $ $ < = MMMMMMM M`7#