Ukunyanzeliswa kweNgqwalasela yeFlash kwi-TPU kunye nokufunda ngeNdlela eNzima

Ukunyanzeliswa kweNgqwalasela yeFlash kwi-TPU kunye nokufunda ngeNdlela eNzima

Usukelo lokuphucula yingoma yesiren yeenjineli. Ayithembisi nje inzuzo eyongeziweyo, kodwa uyolo lokugoba ihardware kwintando yakho. I-odyssey yam yakutsha nje ekunyanzeliseni ukuphunyezwa kweFlash Attention ye-state-of-the-art eyenzelwe i-NVIDIA GPUs-kwi-TPU kaGoogle yazalwa kulo mtsalane. Injongo yayincomeka: ukukhawulezisa umbhobho obalulekileyo wokuqonda. Uhambo, nangona kunjalo, yayiludidi oluphambili kwiinyaniso ezinzima zoyilo lwemodyuli. Libali eligxininisa ukuba kutheni amaqonga afana neMewayz, amkela kwaye alawule ukungafani kwetekhnoloji, eyimfuneko kwimisebenzi yoshishino oluzinzileyo.

Ingoma yeSiren yePeak Performance

Ukuqwalaselwa kweFlash yi-algorithm yohlaziyo olukhawulezisa ngokumangalisayo imifuziselo yeTransformer ngokwandisa ufikelelo kwimemori. KwiiGPUs yayiyilelwe, ngumlingo omsulwa. Isicelo sethu esingundoqo, injini yokucubungula uxwebhu, ixhomekeke kakhulu kule mifuziselo. Ukubona iinombolo zebenchmark, i-equation yabonakala ilula: Ingqalelo yeFlash + yethu i-TPU quota = ukusetyenzwa ngokukhawuleza kunye neendleko eziphantsi. Ndingene ngaphakathi, ndiqinisekile ukuba ngokwanelisayo kwinqanaba eliphantsi lokukhala-ukujijisana ne-kernel layouts, izithuba zememori, kunye nomqokeleli we-XLA-ndinokwenza esi sikwere sesikwere singene kumngxuma ojikelezayo, owenziwe nge-tensor-processing. Ingqwalasela yokuqala ibikuloyiso lobuchule, hayi ukubetha kwentliziyo kwexesha elide.

I-Cascade Yezinto Ezintsonkothileyo Ezingabonakaliyo

Eyokuqala "impumelelo" yayinxilisayo. Emva kweeveki, ndafumana imodeli endiza kuyiqhuba. Kodwa uloyiso lwaba lilize. I-hack yayibuthathaka, iqhekeza lonke uhlaziyo oluncinci lwethala leencwadi. Okubi nakakhulu, yenze ukutsalwa okungabonakaliyo kuwo wonke umbhobho. Indlela yekhowudi ye-TPU ye-bespoke yaba silo, isinyanzela ukuba sigcine izikripthi zokusasazwa ezahlukileyo, iigwegwe zokubeka iliso, kunye nengqiqo yokulayisha idatha. Oko kwakuthethwa ukuba yimodyuli ephuculweyo yaba yibhokisi emnyama emdaka. Siye saba nokusilela okubuhlungu:

• Debugging Hell: Izixhobo zeprofayile ezisemgangathweni beziyimfama kwi-kernel yethu yesiko, zenza uhleko lokusebenza lube liphupha lokufumanisa.
• Iqela le-Bottleneck: Kuphela ndiqonde ikhowudi ye-labyrinthine, ndimisa uphuhliso ukuba andifumaneki.
• Ityala loHlanganiso: Uphuculo oluphezulu kwimodeli ephambili ayikwazanga kufakwa lula kwifolokhwe yethu ye-TPU ye-frankenstein.
• Iindleko zeZikhukula: Inkumbulo evuzayo engaqondakaliyo kwi-TPU, ezelwe kulawulo lwethu lwenkumbulo engaqhelekanga, yakha yakhokelela ekugqithiseni kweendleko ezingama-40% ngaphambi kokuba siyibambe.

I-Modular Mindset: Udibaniso ngaphaya kokuFaka ukunyanzeliswa

Esona sifundo besingengo TPUs okanye ii-algorithms zokuhoya. Yayimalunga nokumodareyitha. Saphule umthetho-siseko osisiseko: amalungu enkqubo kufuneka atshintshe kwaye asebenzisane, angadityaniswa kunye. Ngokunyanzelisa icandelo elingeyiyo inzalelwane kwisitaki sethu, siye sancama uzinzo, ukucaca, kunye nobuchule bokusebenza okuphezulu okucingelwayo okungazange kubonwe kwimveliso. Kulapho ifilosofi ye-OS yeshishini lemodyuli efana neMewayz iba baluleke kakhulu. I-Mewayz ayithethi malunga nokukuvalela kwisitaki esinye; imalunga nokubonelela ngomaleko we-orchestration okuvumela ukuba usebenzise esona sixhobo sifanelekileyo somsebenzi-inokuba yi-GPU-ingqwalasela ethile okanye imodeli ye-TPU-ngaphandle kokwakha nokugcina isihlunu esidibanisayo ngokwakho.

"Ukwandisa okwandisa ukuntsokotha kwenkqubo kudla ngokuba lityala lobugcisa elizayo elifihliweyo njengenkqubela phambili. Ubuchule bokwenyani buvela kujongano olucocekileyo kunye neendawo ezinokutshintshwa, hayi ubuqhawe obudityanisiweyo obunye."

Ukufunda kunye nokuJikela kwisantya esiZinzileyo

Ekugqibeleni siye saluvala umfuniselo olunyanzelweyo lweFlash Attention. Endaweni yoko, siye sanikela ingqalelo ekuphunyezweni kwengqwalasela ye-TPU eyathi, ngelixa ithiyori icotha ephepheni, yabonakala ithembekile kwaye inokugcinwa. Isixokelelwano siphela siphuculwe ngenxa yozinzo lwayo. Okubaluleke ngakumbi, saqala ukuyila iinkonzo zethu ze-AI njengeemodyuli ezicacileyo, ezichazwe kakuhle. Olu tshintsho kwindlela yokucinga-ukubeka phambili izivumelwano ezicocekileyo phakathi kwamacandelo ngaphezu kokusebenza okukrwada, kwendawo-yinto kanye evumela amashishini ukuba akhule ngobukrelekrele. Kwihlabathi lezixhobo eziguqukayo ngokukhawuleza, iqonga elifana neMewayz libonelela ngesakhelo sokuplaga kwizakhono ezitsha ngaphandle kokwakha kwakhona ivili, okanye kwimeko yethu, ngaphandle kokuzama ukubuyisela iprosesa. Indlela enzima yasifundisa ukuba isantya esizinzileyo asikokuphumelela kwidabi ngalinye elincinci, kodwa kukuqinisekisa ukuba wonke umkhosi wakho unokumatsha ngaxeshanye.

Imibuzo Ebuzwa Rhoqo

Ukunyanzeliswa kweNgqwalasela yeFlash kwi-TPU kunye nokufunda ngeNdlela eNzima

Usukelo lokuphucula yingoma yesiren yeenjineli. Ayithembisi nje inzuzo eyongeziweyo, kodwa uyolo lokugoba ihardware kwintando yakho. I-odyssey yam yakutsha nje ekunyanzeliseni ukuphunyezwa kweFlash Attention ye-state-of-the-art eyenzelwe i-NVIDIA GPUs-kwi-TPU kaGoogle yazalwa kulo mtsalane. Injongo yayincomeka: ukukhawulezisa umbhobho obalulekileyo wokuqonda. Uhambo, nangona kunjalo, yayiludidi oluphambili kwiinyaniso ezinzima zoyilo lwemodyuli. Libali eligxininisa ukuba kutheni amaqonga afana neMewayz, amkela kwaye alawule ukungafani kwetekhnoloji, eyimfuneko kwimisebenzi yoshishino oluzinzileyo.

Ingoma yeSiren yePeak Performance

Ukuqwalaselwa kweFlash yi-algorithm yohlaziyo olukhawulezisa ngokumangalisayo imifuziselo yeTransformer ngokwandisa ufikelelo kwimemori. KwiiGPUs yayiyilelwe, ngumlingo omsulwa. Isicelo sethu esingundoqo, injini yokucubungula uxwebhu, ixhomekeke kakhulu kule mifuziselo. Ukubona iinombolo zebenchmark, i-equation yabonakala ilula: Ingqalelo yeFlash + yethu i-TPU quota = ukusetyenzwa ngokukhawuleza kunye neendleko eziphantsi. Ndingene ngaphakathi, ndiqinisekile ukuba ngokwanelisayo kwinqanaba eliphantsi lokukhala-ukujijisana ne-kernel layouts, izithuba zememori, kunye nomqokeleli we-XLA-ndinokwenza esi sikwere sesikwere singene kumngxuma ojikelezayo, owenziwe nge-tensor-processing. Ingqwalasela yokuqala ibikuloyiso lobuchule, hayi ukubetha kwentliziyo kwexesha elide.

I-Cascade yeeNgxaki ezingabonakaliyo

Eyokuqala "impumelelo" yayinxilisayo. Emva kweeveki, ndafumana imodeli endiza kuyiqhuba. Kodwa uloyiso lwaba lilize. I-hack yayibuthathaka, iqhekeza lonke uhlaziyo oluncinci lwethala leencwadi. Okubi nakakhulu, yenze ukutsalwa okungabonakaliyo kuwo wonke umbhobho. Indlela yekhowudi ye-TPU ye-bespoke yaba silo, isinyanzela ukuba sigcine izikripthi zokusasazwa ezahlukileyo, iigwegwe zokubeka iliso, kunye nengqiqo yokulayisha idatha. Oko kwakuthethwa ukuba yimodyuli ephuculweyo yaba yibhokisi emnyama emdaka. Siye saba nokusilela okubuhlungu:

I-Modular Mindset: Udibaniso ngaphaya kokuFaka ukunyanzeliswa

Esona sifundo besingengo TPUs okanye ii-algorithms zokuhoya. Yayimalunga nokumodareyitha. Saphule umthetho-siseko osisiseko: amalungu enkqubo kufuneka atshintshe kwaye asebenzisane, angadityaniswa kunye. Ngokunyanzelisa icandelo elingeyiyo inzalelwane kwisitaki sethu, siye sancama uzinzo, ukucaca, kunye nobuchule bokusebenza okuphezulu okucingelwayo okungazange kubonwe kwimveliso. Kulapho ifilosofi ye-OS yeshishini lemodyuli efana neMewayz iba baluleke kakhulu. I-Mewayz ayithethi malunga nokukuvalela kwisitaki esinye; imalunga nokubonelela ngomaleko we-orchestration okuvumela ukuba usebenzise esona sixhobo sifanelekileyo somsebenzi-inokuba yi-GPU-ingqwalasela ethile okanye imodeli ye-TPU-ngaphandle kokwakha nokugcina isihlunu esidibanisayo ngokwakho.

Ukufunda kunye nokuJikela kwisantya esiZinzileyo

Ekugqibeleni siye saluvala umfuniselo olunyanzelweyo lweFlash Attention. Endaweni yoko, siye sanikela ingqalelo ekuphunyezweni kwengqwalasela ye-TPU eyathi, ngelixa ithiyori icotha ephepheni, yabonakala ithembekile kwaye inokugcinwa. Isixokelelwano siphela siphuculwe ngenxa yozinzo lwayo. Okubaluleke ngakumbi, saqala ukuyila iinkonzo zethu ze-AI njengeemodyuli ezicacileyo, ezichazwe kakuhle. Olu tshintsho kwindlela yokucinga-ukubeka phambili izivumelwano ezicocekileyo phakathi kwamacandelo ngaphezu kokusebenza okukrwada, kwendawo-yinto kanye evumela amashishini ukuba akhule ngobukrelekrele. Kwihlabathi lezixhobo eziguqukayo ngokukhawuleza, iqonga elifana neMewayz libonelela ngesakhelo sokuplaga kwizakhono ezitsha ngaphandle kokwakha kwakhona ivili, okanye kwimeko yethu, ngaphandle kokuzama ukubuyisela iprosesa. Indlela enzima yasifundisa ukuba isantya esizinzileyo asikokuphumelela kwidabi ngalinye elincinci, kodwa kukuqinisekisa ukuba wonke umkhosi wakho unokumatsha ngaxeshanye.