Әh L ddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZmZmZmZgdZe dZe dd ZeeefZeed fZe d eeZGd deeZGddeee eZGddeeed fZGddeeZ GddeeZ!GddeZ"GddeeZ#efdeedee e$e%fde ede&e#efdZ'dS) N)Sequence)castGenericIterableOptionalTypeVarUnion) deprecated)default_generator GeneratorrandpermTensor)DatasetIterableDataset TensorDataset StackDataset ConcatDataset ChainDatasetSubset random_split_T_T_coT) covariant._T_stackc&eZdZdZdefdZd dZdS) raAn abstract class representing a :class:`Dataset`. All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite :meth:`__getitem__`, supporting fetching a data sample for a given key. Subclasses could also optionally overwrite :meth:`__len__`, which is expected to return the size of the dataset by many :class:`~torch.utils.data.Sampler` implementations and the default options of :class:`~torch.utils.data.DataLoader`. Subclasses could also optionally implement :meth:`__getitems__`, for speedup batched samples loading. This method accepts list of indices of samples of batch and returns list of samples. .. note:: :class:`~torch.utils.data.DataLoader` by default constructs an index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided. returnc td)Nz3Subclasses of Dataset should implement __getitem__.)NotImplementedErrorselfindexs `/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/torch/utils/data/dataset.py __getitem__zDataset.__getitem__:s!"WXXXotherDataset[_T_co]ConcatDataset[_T_co]c$t||gSN)rr r%s r"__add__zDataset.__add__AsdE]+++r$N)r%r&rr')__name__ __module__ __qualname____doc__rr#r+r$r"rr'sR$YEYYYY,,,,,,r$rc*eZdZdZdeefdZdS)raIAn iterable Dataset. All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream. All subclasses should overwrite :meth:`__iter__`, which would return an iterator of samples in this dataset. When a subclass is used with :class:`~torch.utils.data.DataLoader`, each item in the dataset will be yielded from the :class:`~torch.utils.data.DataLoader` iterator. When :attr:`num_workers > 0`, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. :func:`~torch.utils.data.get_worker_info`, when called in a worker process, returns information about the worker. It can be used in either the dataset's :meth:`__iter__` method or the :class:`~torch.utils.data.DataLoader` 's :attr:`worker_init_fn` option to modify each copy's behavior. Example 1: splitting workload across all workers in :meth:`__iter__`:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER) >>> # xdoctest: +SKIP("Fails on MacOS12") >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example code only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... worker_info = torch.utils.data.get_worker_info() ... if worker_info is None: # single-process data loading, return the full iterator ... iter_start = self.start ... iter_end = self.end ... else: # in a worker process ... # split workload ... per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... iter_start = self.start + worker_id * per_worker ... iter_end = min(iter_start + per_worker, self.end) ... return iter(range(iter_start, iter_end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [tensor([3]), tensor([4]), tensor([5]), tensor([6])] >>> # xdoctest: +REQUIRES(POSIX) >>> # Multi-process loading with two worker processes >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> # xdoctest: +IGNORE_WANT("non deterministic") >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [tensor([3]), tensor([5]), tensor([4]), tensor([6])] >>> # With even more workers >>> # xdoctest: +IGNORE_WANT("non deterministic") >>> print(list(torch.utils.data.DataLoader(ds, num_workers=12))) [tensor([3]), tensor([5]), tensor([4]), tensor([6])] Example 2: splitting workload across all workers using :attr:`worker_init_fn`:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER) >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example code only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... return iter(range(self.start, self.end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [3, 4, 5, 6] >>> >>> # Directly doing multi-process loading yields duplicate data >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [3, 3, 4, 4, 5, 5, 6, 6] >>> # Define a `worker_init_fn` that configures each dataset copy differently >>> def worker_init_fn(worker_id): ... worker_info = torch.utils.data.get_worker_info() ... dataset = worker_info.dataset # the dataset copy in this worker process ... overall_start = dataset.start ... overall_end = dataset.end ... # configure the dataset to only process the split workload ... per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... dataset.start = overall_start + worker_id * per_worker ... dataset.end = min(dataset.start + per_worker, overall_end) ... >>> # Mult-process loading with the custom `worker_init_fn` >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn))) [3, 5, 4, 6] >>> # With even more workers >>> print(list(torch.utils.data.DataLoader(ds, num_workers=12, worker_init_fn=worker_init_fn))) [3, 4, 5, 6] r%c$t||gSr))rr*s r"r+zIterableDataset.__add__sT5M***r$N)r,r-r.r/rrr+r0r$r"rrIs@jjX+WU^++++++r$rcJeZdZUdZeedfed<deddfdZdZdZ dS) rzDataset wrapping tensors. Each sample will be retrieved by indexing tensors along the first dimension. Args: *tensors (Tensor): tensors that have the same size of the first dimension. .tensorsrNc`tfdDs Jd|_dS)Nc3|K|]6}dd|dkV7dS)rN)size).0tensorr4s r" z)TensorDataset.__init__..sS  5;GAJOOA  &++a.. 0      r$zSize mismatch between tensors)allr4)r r4s `r"__init__zTensorDataset.__init__s_    ?F      + + * + + + r$cDtfd|jDS)Nc3(K|] }|V dSr)r0)r8r9r!s r"r:z,TensorDataset.__getitem__..s'>>vVE]>>>>>>r$)tupler4rs `r"r#zTensorDataset.__getitem__s(>>>>>>>>>>r$cB|jddSNr)r4r7r s r"__len__zTensorDataset.__len__s|A##A&&&r$) r,r-r.r/r?r__annotations__r<r#rCr0r$r"rrsw63; D ???'''''r$rcreZdZUdZeeefed<dee dee ddfdZ dZ d e fd Z d ZdS) raDataset as a stacking of multiple datasets. This class is useful to assemble different parts of complex input data, given as datasets. Example: >>> # xdoctest: +SKIP >>> images = ImageDataset() >>> texts = TextDataset() >>> tuple_stack = StackDataset(images, texts) >>> tuple_stack[0] == (images[0], texts[0]) >>> dict_stack = StackDataset(image=images, text=texts) >>> dict_stack[0] == {'image': images[0], 'text': texts[0]} Args: *args (Dataset): Datasets for stacking returned as tuple. **kwargs (Dataset): Datasets for stacking returned as dict. datasetsargskwargsrNc|r^|rtdt|d_tfd|Drtd|_dS|rnt |}t|d_tfd|Drtd|_dStd)NztSupported either ``tuple``- (via ``args``) or``dict``- (via ``kwargs``) like input/output, but both types are given.rc3HK|]}jt|kVdSr)_lengthlenr8datasetr s r"r:z(StackDataset.__init__..s0DDG4<3w<</DDDDDDr$zSize mismatch between datasetsc3HK|]}jt|kVdSr)rKrNs r"r:z(StackDataset.__init__..s0CCG4<3w<</CCCCCCr$z%At least one dataset should be passed) ValueErrorrMrLanyrFlistvalues)r rGrHtmps` r"r<zStackDataset.__init__s  F  ^tAw<z,StackDataset.__getitem__..s#NNN*!WAwu~NNNr$c3(K|] }|V dSr)r0)r8rOr!s r"r:z+StackDataset.__getitem__..s'AAWU^AAAAAAr$) isinstancerFdictitemsr?rs `r"r#zStackDataset.__getitem__se dmT * * ONNNN 8K8K8M8MNNN NAAAA4=AAAAAAr$indicesc t|jtrd|D}|jD]\}}t t |ddr||}t|t|kr/tdt|dt|t||D] \}}|||< t||D]\}}||||<|Sd|D} |jD]}t t |ddr||}t|t|kr/tdt|dt|t|| D]\}} | |t|| D] \}} | ||!d| D} | S)Ncg|]}iSr0r0r8_s r" z-StackDataset.__getitems__..s(=(=(=(=(=(=r$ __getitems__z0Nested dataset's output size mismatch. Expected z, got cg|]}gSr0r0ras r"rcz-StackDataset.__getitems__..s!6!6!6"!6!6!6r$c,g|]}t|Sr0)r?)r8samples r"rcz-StackDataset.__getitems__..%s&N&N&NuV}}&N&N&Nr$) r[rFr\r]callablegetattrrdrMrQzipappend) r r^ dict_batchrXrOr]datad_sampleidx list_batcht_sample tuple_batchs r"rdzStackDataset.__getitems__sw dmT * * (=(=W(=(=(=J"m1133 3 3 7GG^TBBCC 3#0099E5zzS\\11(J),WJJ=@ZZJJ+.eZ*@*@++h&* +*-Wj)A)A33 X&-cl 3 "7!6g!6!6!6 } 2 2G>>?? 2,,W55u::W--$F%(\\FF9`cummulative_sizes` attribute is renamed to `cumulative_sizes`)categoryc|jSr)rrBs r"cummulative_sizeszConcatDataset.cummulative_sizes\s $$r$)r,r-r.r/rSrrrDint staticmethodr|rr<rCr#propertyr FutureWarningr __classcell__rs@r"rr,s75>""""3i\;'!2;t;;;;;;))) 6 6 6ZH%% X %%%%%r$rcDeZdZdZdeeddffd ZdZdZxZ S)ra_Dataset for chaining multiple :class:`IterableDataset` s. This class is useful to assemble different existing dataset streams. The chaining operation is done on-the-fly, so concatenating large-scale datasets with this class will be efficient. Args: datasets (iterable of IterableDataset): datasets to be chained together rFrNcVt||_dSr))r~r<rF)r rFrs r"r<zChainDataset.__init__ps$   r$c#lK|jD])}t|ts Jd|Ed{V*dS)N*ChainDataset only supports IterableDataset)rFr[r)r rs r"__iter__zChainDataset.__iter__tsd  A? < <; < < <LLLLLLLL   r$cd}|jD]3}t|ts Jd|t|z }4|S)Nrr)rFr[rrM)r totalrs r"rCzChainDataset.__len__{s^  A? < <; < < < SVVOEE r$) r,r-r.r/rrr<rrCrrs@r"rresx!'!2!t!!!!!!r$rceZdZUdZeeed<eeed<deedeeddfdZ dZ de ede efdZ d Z dS) rz Subset of a dataset at specified indices. Args: dataset (Dataset): The whole Dataset indices (sequence): Indices in the whole set selected for subset rOr^rNc"||_||_dSr)rOr^)r rOr^s r"r<zSubset.__init__s  r$ct|trjfd|DSjj|S)Nc*g|]}j|Sr0r^)r8ir s r"rcz&Subset.__getitem__..s > > >Qa > > >r$)r[rSrOr^)r ros` r"r#zSubset.__getitem__sL c4  @< > > > ># > > >? ?|DL-..r$cttjddr&jfd|DSfd|DS)Nrdc*g|]}j|Sr0rr8ror s r"rcz'Subset.__getitems__..s -S-S-SCdl3.?-S-S-Sr$c@g|]}jj|Sr0rrs r"rcz'Subset.__getitems__..s'GGGDLc!23GGGr$)rhrirOrd)r r^s` r"rdzSubset.__getitems__si GDL.$?? @ @ H<,,-S-S-S-S7-S-S-STT TGGGGwGGG Gr$c*t|jSr))rMr^rBs r"rCzSubset.__len__s4<   r$)r,r-r.r/rrrDrrr<r#rSrdrCr0r$r"rrsU^ c]#4/// HDIH$u+HHHH!!!!!r$rrOlengths generatorrc tjt|drt|dkrg}t|D]j\}}|dks|dkrt d|dt tjt|z}||ktt|z }t|D]$}|t|z}||xxdz cc<%|}t|D]#\}} | dkrtj d|d$t|tkrt dtt|| ttt |} fd t!t#j||DS) a Randomly split a dataset into non-overlapping new datasets of given lengths. If a list of fractions that sum up to 1 is given, the lengths will be computed automatically as floor(frac * len(dataset)) for each fraction provided. After computing the lengths, if there are any remainders, 1 count will be distributed in round-robin fashion to the lengths until there are no remainders left. Optionally fix the generator for reproducible results, e.g.: Example: >>> # xdoctest: +SKIP >>> generator1 = torch.Generator().manual_seed(42) >>> generator2 = torch.Generator().manual_seed(42) >>> random_split(range(10), [3, 7], generator=generator1) >>> random_split(range(30), [0.3, 0.3, 0.4], generator=generator2) Args: dataset (Dataset): Dataset to be split lengths (sequence): lengths or fractions of splits to be produced generator (Generator): Generator used for the random permutation. rrzFraction at index z is not between 0 and 1zLength of split at index z- is 0. This might result in an empty dataset.zDSum of input lengths does not equal the length of the input dataset!)rcLg|] \}}t||z |!Sr0)r)r8offsetlengthrOr^s r"rcz random_split..sE    FF w& 89::   r$)mathisclosesum enumeraterQrfloorrMrkrangewarningswarnr tolistrrrj itertools accumulate) rOrrsubset_lengthsrfracn_items_in_split remainder idx_to_add_atrr^s ` @r"rrs< |CLL!$$W):):$& )) 4 4GAtaxx4!88 !Pa!P!P!PQQQ" 3w<<$.//     ! !"2 3 3 3 3LL3~#6#66 y!! / /AN 3 33M = ) ) )Q . ) ) ) ) "7++  IAv{{ >>>>  7||s7||## R   s7||y999@@BBG8C='**G     !)"6w"?"?II   r$)(rrrrcollections.abcrtypingrrrrrr typing_extensionsr torchr r r r__all__rrr\str_T_dictr?_T_tuplerrrrrrrrrfloatrSrr0r$r"rs  $$$$$$EDDDDDDDDDDDDDDD((((((A@@@@@@@@@@@    WT]]4((( sEz    7:x 1 1,,,,,gen,,,Dn+n+n+n+n+genhuon+n+n+h'''''GE&#+./'''0TTTTT78$TTTn6%6%6%6%6%GEN6%6%6%r?@!!!!!WU^!!!H&7?? R[? eCJ' (? "? &* ??????r$