%ɷiddlZddlZddlZddlZddlmZddlmZddlZ ddl Z ddl m Z m Z ddlmZddlmZmZddlmZddlmZmZmZdd lmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&e"rd Z'd Z(nd Z'd Z(d%d Z)d Z*dZ+dZ,dZ-dZ.dZ/dZ0dZ1dZ2dZ3dZ4dZ5dZ6d%dZ7d%dZ8dZ9dZ:dZ;dZd"Z?d#Z@eAd$k(re@yy)&N)deepcopy)Path) DataLoaderDataset) Accelerator)SeedableRandomSamplerprepare_data_loader)AcceleratorState)RegressionDatasetRegressionModelare_the_same_tensors) DataLoaderConfigurationDistributedTypegather gather_objectis_bf16_availableis_cuda_availableis_datasets_availableis_fp16_availableis_hpu_availableis_mps_availableis_pytest_availableset_seedsynchronize_rng_statesgMbP?gư>cn|r%t||t|}t|||St||d|S)zBCreates a dataloader that can also use the `SeedableRandomSampler`) generator data_source num_samples) batch_sizesamplerT)rshuffler)rlenr) train_setrruse_seedable_samplerr s \/var/www/html/venv/lib/python3.12/site-packages/accelerate/test_utils/scripts/test_script.pygenerate_baseline_dataloaderr&9sA(!I  ) GLL) DT]^^c4td|jy)NzPrinting from the main process print process_indexstates r% print_mainr.H +E,?,?+@ ABr'c4td|jy)Nz%Printing from the local main process )r*local_process_indexr,s r%print_local_mainr2Ls 1%2K2K1L MNr'c4td|jy)NPrinting from the last process r)r,s r% print_lastr5Pr/r'c:td|d|jy)NPrinting from process : r))r- process_idxs r%print_onr:Ts ";-r%2E2E1F GHr'c t}|j}td}|j5|jrr#|j3j5dk(s%J|j3j5dk(sJ|j7d|j9dt)j*|5|jAtB|j0ddd|jDr=|j3j5d|j0jdz k(s%J|j3j5dk(sJ|j7d|j9dtG|D]}t)j*|5|jItJ||j0|ddd|jL|k(r3|j3j5d|d|jLk(s%J|j3j5dk(sJ|j7d|j9dy#1swYaxYw#1swYnxYw#1swYsxYw#1swY1xYw#t$r|j!wxYw#1swYHxYw#1swYxYw#1swYxYw#1swY xYw)Nzcheck_main_process_first.txt皙?za+zCurrently in the main process zNow on another process zMain process was not firstzOnly wrote to file zNow on another processz times, not z Printing from the main process 0z$ != Printing from the main process 0z != ""rz&Printing from the local main process 0r4)r+r7r8)'r num_processesrmain_process_firstis_main_processtimesleepopenwritewait_for_everyonejoin readlines startswithendswithcountAssertionErrorunlinkexistsioStringIO contextlibredirect_stdouton_main_processr.r-getvaluerstriptruncateseekon_local_main_processr2is_local_main_processon_last_processr5is_last_processrange on_processr:r+) acceleratorr?pathftextresultr9s r%process_execution_checkrcXs-K--M . /D  ' ' )4  & & JJsOdD! ;Q9: ; ;dD! 4Q23 4 4!!#"" $Z *1771;;=)D * ??#DE cGc cEq }}%?@^B^^@::89[=V=VYZ=ZZ %djj1I&JQ&N%O|\g\u\u[vw Z""t{{} !!# A  # #A &C/ ##J/ 0A0ABC ZZ\ "F"";;lxGk=ll;zz|""$*=vhf,==*JJqMFF1I  # #A &O; ))*:;K?mAmm?   ?#<#< <w'#EII$;$;$=>l@ll>!IK=I>  3 3 5 6j8jj 6   A% -.&r'ct}d|jz}tt|d}t ||j |j|j d}g}|D]}|jt|tj|}tj|jtjd|jsJdtt|d}t ||j |j|j dd}g}|D]}|jt|tj|}tj|jtjd|jsJd|j dk(r td tt|dd }t ||j |j|j d}g}|D]}|jt|tj|j!}|j#|t%t|k(sJd tt|dd }t ||j |j|j dd}g}|D]}|jt|tj|j!}|j#|t%t|k(sJd |j&dk(r td yy) N rT) put_on_devicer%Wrong non-shuffled dataloader result.)ru split_batchesz Non-shuffled dataloader passing.rr!!Wrong shuffled dataloader result.zShuffled dataloader passing.r r?rr\r devicer+appendrrgcatequalcpuarangelongr*tolistsortlistr1r-lengthdlrbbatchs r%dl_preparation_checkrs  E %%% %F E&Ma 0B Ru/B/BEDWDWgk lB F% fUm$% YYv F ;;vzz|U\\!V%<%A%A%C DmFmm D E&Ma 0B       BF% fUm$% YYv F ;;vzz|U\\!V%<%A%A%C DmFmm D a 01 E&Ma >B Ru/B/BEDWDWgk lB F% fUm$% YYv  % % 'F KKM T%-( (M*MM ( E&Ma >B       BF% fUm$% YYv  % % 'F KKM T%-( (M*MM (   A% ,-&r'c t}d|jz}tt|d}t ||j |j|j dd}g}|D]}|jt|tj|}tj|jtjd|jsJdtt|d}t ||j |j|j ddd}g}|D]}|jt|tj|}tj|jtjd|jsJd|j dk(r td tt|dd }t ||j |j|j dd}g}|D]}|jt|tj|j!}|j#|t%t|k(sJd tt|dd }t ||j |j|j ddd}g}|D]}|jt|tj|j!}|j#|t%t|k(sJd |j&dk(r td yy) NrrrsrtT)rudispatch_batchesrrv)rurwrz(Non-shuffled central dataloader passing.rxryz$Shuffled central dataloader passing.rzrs r%central_dl_preparation_checkrs  E %%% %F E&Ma 0B  ELL%--u/B/BRVim BF% fUm$% YYv F ;;vzz|U\\!V%<%A%A%C DmFmm D E&Ma 0B      BF% fUm$% YYv F ;;vzz|U\\!V%<%A%A%C DmFmm D a 89 E&Ma >B  ELL%--u/B/BRVim BF% fUm$% YYv  % % 'F KKM T%-( (M*MM ( E&Ma >B      BF% fUm$% YYv  % % 'F KKM T%-( (M*MM (   A% 45&r'ct}Gddt}Gdd}|td|jz}|t |d}t ||}t ||j|j|j}t|jd r(t|jj|sJd yt|j|sJd y) NceZdZdZdZdZy)+custom_sampler_check..CustomDatasetc||_yNdata)selfrs r%__init__z4custom_sampler_check..CustomDataset.__init__<s DIr'c,t|jSr)r"rrs r%__len__z3custom_sampler_check..CustomDataset.__len__?styy> !r'c |j|Srr)rindexs r% __getitem__z7custom_sampler_check..CustomDataset.__getitem__Bs99U# #r'N)__name__ __module__ __qualname__rrrr'r% CustomDatasetr;s  " $r'rc.eZdZddededefdZdZdZy) 0custom_sampler_check..CustomBatchSamplerdataset_lengthrr!cT||_tj||_||_yr)rnpr data_indexr!)rrrr!s r%rz9custom_sampler_check..CustomBatchSampler.__init__Fs (DO ii7DO"DLr'c3Kt|}|jr*tjj |j }n |j }tj ||}|Ed{y7wr)r"r!rrandom permutationr array_split)r num_batchesroutputs r%__iter__z9custom_sampler_check..CustomBatchSampler.__iter__KsPd)K|| --doo>^^E;7F  sA)A3+A1,A3cltjt|j|jz Sr)mathceilr"rrrs r%rz8custom_sampler_check..CustomBatchSampler.__len__Ts#99S1DOOCD Dr'N)T)rrrintboolrrrrr'r%CustomBatchSamplerrEs' #3 #C #$ #   Er'rrrrsrt) batch_samplerrz>Custom sampler was changed after calling `prepare_data_loader`) r rr\r?r"rr r{r+hasattrr isinstance)r-rrdatasetr rs r%custom_sampler_checkr8s  E$$EE$E"u':':":;? L ?r'ctdtdd}t|dd}td}t |}|j |}g}t d D]}|D]}|j|d tj|}td|jd g}t d D]}|D]}|j|d tj|}tj||sJd y) N* rseedTrxr$dataloader_configxrz;Did not obtain the same items with the same seed and epoch.) rr rrrpreparer\r|rgr} set_epochallclose)r#train_dlconfigr^original_items_r new_itemss r%check_seedable_samplerrfs  RL!"5I)4@H $$ ?F7K""8,HN 1X. .E  ! !%* - ..YY~.N RL qI 1X) )E   U3Z ( )) )$I >>.) 4s6ss 4r'c>tdtd}t|}|jdkDsJdt t t ddd}t|d }|jjj}t|tsJd y) NrTrrr>z)This test requires more than one process.rrx) dataloaderr$z:Sampler in BatchSamplerShard is not SeedableRandomSampler.) rrrr?rrr\r rr rr)rr^rprepared_data_loadertarget_samplers r%-check_seedable_sampler_in_batch_sampler_shardrs RL $$ ?F7K  $ $q (U*UU (DrO4HJ.! *77EEMMN n&; <D Did not obtain the same model on CPU or distributed training. rmsgs r%z training_check..YZ]Y^_r')atolrtolrc d|Srrrs r%rz training_check..rr'zVTraining yielded the same results on one CPU or distributed setup with no batch split.T)rwr$rc d|Srrrs r%rz training_check..rr'c d|Srrrs r%rz training_check.. rr'zSTraining yielded the same results on one CPU or distributed setup with batch split.zKeep fp32 wrapper check.fp16)mixed_precisionr)keep_fp32_wrapperr>)dtyper{zBF16 training check.rbf16)rrc d|Srrrs r%rz training_check..>]^a]bcr'c d|Srrrs r%rz training_check..Err'zFP16 training check.c d|Srrrs r%rz training_check..drr'c d|Srrrs r%rz training_check..krr')+r rgrnr?rr abrr&r rrrrrrr\rrrrrr unwrap_modelrtesting assert_closeATOLRTOLr*rrr _reset_stateLinearTensortofloat16r{rrr)r$r-rrrr# old_modelr^rrrrrrrrmodel_with_fp32_wrapper input_tensors r%training_checkrs9  E!IJ !^e11 1F(e>Q>Q1QS\^rsIy  ,`.`` ,  ,`.`` ,-K+Iy*NbcH  E  0 0 2s;I!,!4!4Xui!PHeY RL " 1X E OO 5:&F88&&//c CD   & NN     $ $U + / / 1E MM     _  MM     _ no/dYmn0ABK+9j5+>+>>@TH  E  0 0 2s;I!,!4!4Xui!PHeY RL " 1X E OO 5:&F88&&//c CD   & NN     $ $U + / / 1E MM     _  MM     _ kl.0 ()%%'!&9 1%##E*"-":":5TX":"Y||QF+..U]];K]K].^ (6 $%%%'3I]^!&L]^ / 9jRfg!KKOOE$4$4$63O? %0%8%85)%T"% b!q !A! !!uSz*xx**33FE#JG$$T*  ! !((/335 "" KK GGc #   "" KK GGc # #3#5 $%%%'3I]^!&L]^ / 9jRfg!KKOOE$4$4$63O? %0%8%85)%T"% b!q !A! !!uSz*xx**33FE#JG$$T*  ! !((/335 "" KK GGc #   "" KK GGc # ;$6r'c `t}|jtd|jzDcgc]}t |c}}|j |d5}t |dk(s Jd|jdt | ddd|jtd|jzdz Dcgc]}t |c}}|j |d5}|jr.t |dk(sNJd|jdt |t |dk(s Jd |jdt |ddd|j|jtd|jzdz Dcgc]}t |c}}|jtd|jzDcgc]}t |c}}||fD]}|d }|j |d 5}|jdk(r7t |t |k(sNJd |jdt |t |dk(s Jd|jdt |g}|D]} |j|  ddd|jt} | dt |D cgc]} | d  } } || k(rJd |d| ycc}w#1swYkxYwcc}w#1swYxYwcc}wcc}w#1swYxYwcc} w)Nr)kF apply_padding4Each process did not have two items. Process index: ; Length: r>z;Last process did not receive a single item. Process index: zLOne of the intermediate processes did not receive two items. Process index: rTz9Single process did not receive all items. Process index: )Gathered results is incorrect. Expected: ; Got: ) r from_listr\r?dictsplit_between_processesr"r+r[rFr|r) datasets_Datasetr-vrresultsodd_data even_dataexpected_outputresults_per_processrbgathered_resultsrrs r%$test_split_between_processes_datasetr%oso  E  % %%EDWDW@W:X&YQtay&Y ZD  & &t5 & A W7|q  B5CVCVBWWabefmbnao p    % %%EDWDW@WZ[@[:\&]Qtay&] ^D  & &t5 & AW  w<1$ MeNaNaMbblmpqxmylz{ $w<1$ ^_d_r_r^ss}BCJK~LM $  ))eAH[H[D[^_D_>`*a4!9*abH **uQI\I\E\?]+^!D1I+^_I9%ws)  * *4t * D 3""a'7|s4y0OPUPcPcOddnorszo{n|}07|q(J5K^K^J__ijmnujviwx(#% ! 3#**62 3 3 !()<="2;SY"?@Q!C&@@&(v,UVeUffmntmu*vv(-w+'Z  '^+b+^ 3 3"AsCK1/K6=L*A)LLL BL L+6LLL( c t}ttdd|jz}|j |5}t |dk(s Jd|j dt | ddd|jttdd|jz}ttdd|jzdz }||fD]}|}|j |d5}tjt ||jz }t ||k(s-Jd|jd |j dt |g}|D]}|j| ddd|jt} | dt |} || k(rJd |d | y#1swYExYw#1swYTxYw) Nrrrrr>TrProcess ; did not get the correct number of item(s). Process index: rr) r rr\r?rr"r+rFrrr{r|r) r-rrr rr!num_samples_per_devicer"rbr#rs r%!test_split_between_processes_listr*s  E aU0001 2D  & &t , 7|q  B5CVCVBWWabefmbnao p    U1q5#6#6689IE!a%"5"55:;1F;>G ct}gd}gd}tjgd}|jdvr|||d}t |}|j |5}|j dk(r|d|ddd|jzk(sTJ|jdk(r|d|dd dk(s5J|j d k(r$|d|dd dk(sJd |dd d |d|j dk(r|d|ddd|jzk(s@J|jdk(r|d|dd dk(s!J|j d k(r|d|dd dk(sJ|j dk(rOtj|d|ddd|jzsJd|ddd|jzd|d|jdk(r5tj|d|dd dsZJd|dd dd|d|j d k(r5tj|d|dd dsJd|dd dd|dddd|jy#1swYxYw)N)r>rrrrs)rrcder`gh)r>rr)rrr/rrrsrrrz Expected: , Actual: rr/z7Did not obtain expected values on process 0, expected `z `, received: z7Did not obtain expected values on process 2, expected `z7Did not obtain expected values on process 4, expected `) r rgtensorr?rrr+rrF)r-rrr/r data_copyrs r%(test_split_between_processes_nested_dictr8s   E A0A -.A i'QQ'TN  * *4 0 G""a's|y~6PU=P=P8P'QQQQ$$)s|y~ab'9999$$)s|y~bc'::ujSVXZI[H\\fgnorgsft)rr>rr)rr,r-r.rrTrr'r(rrrr)r r?rgr6r r{rr+rrrFr\rrrr"r|r) r-rrexpectedir rr!r)r"rbr#rs r%#test_split_between_processes_tensorr<s  E Q||\<89<BK2K 2K; ct}|jdvrttd}t ||jz}t ||jz}|j |5}|j |kr*t ||dzk(s=Jd|dzdt |t ||k(sJd|dt |ddd|jy#1swYxYw)N)r>rrrsr>z2Each Process should have even elements. Expected: r5)r r?rr\r"rr+rF)r-rnum_samples_per_process num_extrasrs r%#test_split_between_processes_evenlyrAs  E l*E"I"%d)u/B/B"BY!4!44  * *4 0 G""Z/7|'>'BBHI`cdIdHeeopst{p|o}~B7|'>>HI`Haaklopwlxkyz>     s .AC%%C.ct}|jdusJ|jr|j|jdusJ|jdusJy)NFT)r check_triggerrA set_trigger)r^s r% test_triggerrEsi-K  $ $ &% // /""!  $ $ &$ .. .  $ $ &% // /r'cddl}tjtjj dd}t }tj|jt5}|j|ddddtjjdvsJdt|jjdvsJy#1swYVxYw)Nrr>z*`AcceleratorState` object has no attributez1This happens if `AcceleratorState._reset_state()`) pytestr rrgrr rraisesAttributeErrorrstrvalueargs)rG simple_modelr^cms r%test_reinstantiated_staterO+s!!#88??1a(L-K!!# ~ &*"L)* 73rxx}}Q?O;P PP P >#bhhmmTUFVBW WW W**s ,CCct}|j}|jdk(r tdt |j |j tjk(rtjj}n |j}||jk(r|jdk(r tdt|jdk(r tdt|jdk(r tdt!|jdk(r tdt#|jdk(r tdt%|jdk(r tdt'rdd lm}t-|n td |jdk(r td t/|jdk(r td t1|j tj2k7r(t5t7t9t;|jd kDr t=|j tj>k(ry|jdk(r tdtAdtAd|jdk(r tdtCtEr$|jdk(r tdtG|jIy)Nrz**Initialization**z **Test process execution**z+ **Test split between processes as a list**z+ **Test split between processes as a dict**z- **Test split between processes as a tensor**z( **Test split between processes evenly**z7 **Test split between processes as a datasets.Dataset**)rz6Skipped because Hugging Face datasets is not availablez1 **Test random number generator synchronization**z **DataLoader integration test**r>z **Training integration test**FrTz **Breakpoint trigger test**z **Test reinstantiated state**)%rr-r1r*rerFrkrrlrgrh device_countr?r+rcr*r8r<rArdatasetsrr%rprXLArrrrr DEEPSPEEDrrErrOdestroy_process_group)r^r-num_processes_per_noders r%mainrW;sH-K   E   A% "#  !:!::!&!8!8!:!&!4!4!4!44   ! # 0 1!   ! # @ A)+   ! # @ A02   ! # B C+-   ! # = >+-   ! # L M " < 01A B J K   A% BC   A% 12 !4!44$& -/ Q57 !:!::   A% /0.-   A% -.N  $ $ ) 3 4!# !r'__main__)F)BrQrOrrBcopyrpathlibrnumpyrrgtorch.utils.datarr accelerateraccelerate.data_loaderrr accelerate.stater accelerate.test_utilsr r r accelerate.utilsrrrrrrrrrrrrrrrr&r.r2r5r:rcrerprrrrrrrrr%r*r8r<rArErOrWrrr'r%rbs"  0"M-ZZ" D D D D _COCIFR/$9.x>6B+ \t4&n8$k \-w`wB#LwD$0" X N"b zFr'