Okukaka Flash Attention ku TPU n'okuyiga mu ngeri enzibu

"Okulongoosa ekyongera ku buzibu bw'enkola kitera okuba ebbanja ly'eby'ekikugu ery'omu maaso lyokka eryefudde enkulaakulana. Obulung'amu obw'amazima buva mu nkolagana ennyonjo n'ebitundu ebikyusibwa, so si kugatta kwa buzira omulundi gumu."

Okuyiga n’okukyuka okutuuka ku sipiidi ey’olubeerera

Twatuuka ekiseera ne tussa ku shelves okugezesa kwa Flash Attention okukakasibwa. Wabula, twa pivoted ku TPU-native attention implementation nti, wadde nga theoretically slower ku lupapula, eraga wala nnyo okwesigika era okulabirira. Okutwalira awamu enkola y’okuyita mu butuufu yalongooka olw’obutebenkevu bwayo. Ekisinga obukulu, twatandika okuzimba empeereza zaffe eza AI nga modulo ezitali zimu, ezitegeerekese obulungi. Enkyukakyuka eno mu ndowooza —okukulembeza endagaano ennyonjo wakati w’ebitundu okusinga enkola embisi, ey’omu kitundu —kye kyennyini ekisobozesa bizinensi okulinnyisa omutindo mu ngeri ey’amagezi. Mu nsi ya hardware ekulaakulana amangu, platform nga Mewayz egaba framework okussaamu obusobozi obupya nga tetuzzeemu kuzimba nnamuziga, oba mu mbeera yaffe, awatali kugezaako kuddamu kuyiiya processor. Engeri enzibu yatuyigiriza nti sipiidi ey’olubeerera si kuwangula buli lutalo olutonotono, wabula okukakasa nti eggye lyo lyonna lisobola okutambula mu ddoboozi limu.

Ebibuuzo Ebitera Okubuuzibwa

Okukaka Flash Attention ku TPU n'okuyiga mu ngeri enzibu

Okugoberera okulongoosa (optimisation) luyimba lwa siren eri bayinginiya. Tekisuubiza magoba gokka agagenda geeyongera, wabula n’essanyu ly’okugoya hardware okusinziira ku kwagala kwo. Odyssey yange gye buvuddeko mu kukaka enkola ya Flash Attention ey’omulembe —eyakolebwa ku NVIDIA GPUs —ku Google TPU yazaalibwa okuva mu kusikiriza kuno kwennyini. Ekigendererwa kyali kya kitiibwa: okwanguya payipu y’okuteebereza okukulu. Olugendo, wabula, lwali lwa masterclass mu mazima amakakali ag’okukola modular system design. Luno lugero oluggumiza lwaki emikutu nga Mewayz, ezikwata n’okuddukanya obutafaanagana bwa tekinologiya, zeetaagisa nnyo mu nkola ya bizinensi eziwangaala.

Oluyimba lwa Siren olw’okuyimba ku ntikko

Flash Attention ye nkola ey’enkyukakyuka eyanguya nnyo ebika bya Transformer nga elongoosa okuyingira mu jjukira. Ku GPUs ze yategekebwa, bulogo bulongoofu. Enkola yaffe enkulu, yingini ekola ku biwandiiko, yeesigamye nnyo ku bikolwa bino. Okulaba ennamba z’omutindo, ensengekera yalabika ng’ennyangu: Flash Attention + omugabo gwaffe ogwa TPU = okukola amangu n’ebisale ebitono. Nadduka, nga ndi mukakafu nti nga nnina tinkering emala ey’omutindo ogwa wansi —okumegganyizibwa n’ensengeka za kernel, ebifo eby’okujjukira, n’omukung’aanya wa XLA —nsobola okufuula ekikondo kino ekya square okutuuka mu kinnya ekyekulungirivu, ekiringa tensor-processing-shaped. Essira eryasooka lyali liteekebwa ku kuwangula kwa tekinologiya kwokka, so si ku kukuba kw’omutima kw’enkola okw’ekiseera ekiwanvu.

Ekisenge ky’Ebizibu Ebitalabika

"Obuwanguzi" obwasooka bwali butamiiza. Oluvannyuma lwa wiiki, nnafuna omukozi wa firimu gwe nnali ngenda okudduka. Naye obuwanguzi bwali bwa kinnya. Hack yali fragile, nga emenya buli ka library update entonotono. Ekisinga obubi, kyatonda okusika okutalabika ku payipu yonna. Ekkubo lya koodi ya TPU eya bespoke lyafuuka silo, ekyatuwaliriza okukuuma scripts ez’enjawulo ez’okuteeka mu nkola, okulondoola hooks, n’okutuuka ku logic y’okutikka data. Ekyo ekyali kigendereddwamu okuba modulo erongooseddwa kyafuuka brittle black box. Twafuna okulemererwa okuluma:

Endowooza ya Modular: Okugatta ku Force-Fitting

Essomo ery'omusingi teryali ku TPUs oba attention algorithms. Yali ekwata ku modularity. Twali tumenya omusingi omukulu: ebitundu by’enkola birina okuba nga biwanyisiganyizibwa era nga bikolagana, so si ku welding wamu. Nga tukaka ekitundu ekitali kya nzaalwa mu mugogo gwaffe, twasaddaaka okutebenkera, okutegeera, n’obuvumu olw’omutindo ogw’oku ntikko ogw’okuteebereza ogwali tegutera kutuukirira mu kukola. Wano obufirosoofo bwa modular business OS nga Mewayz w’efuuka enkulu. Mewayz si ya kukusibira mu stack emu; kikwata ku kuwa layeri y’okutegeka ekusobozesa okukozesa ekintu ekisinga obulungi ku mulimu —kabeere okulongoosa okukwata ku GPU oba model enzaaliranwa ya TPU —nga tolina kuzimba na kulabirira kitundu kiyunga ggwe kennyini.

Okuyiga n’okukyuka ku sipiidi ey’olubeerera

Twatuuka ekiseera ne tussa ku shelves okugezesa kwa Flash Attention okukakasibwa. Wabula, twa pivoted ku TPU-native attention implementation nti, wadde nga theoretically slower ku lupapula, eraga wala nnyo okwesigika era okulabirira. Okutwalira awamu enkola y’okuyita mu butuufu yalongooka olw’obutebenkevu bwayo. Ekisinga obukulu, twatandika okuzimba empeereza zaffe eza AI nga modulo ezitali zimu, ezitegeerekese obulungi. Enkyukakyuka eno mu ndowooza —okukulembeza endagaano ennyonjo wakati w’ebitundu okusinga enkola embisi, ey’omu kitundu —kye kyennyini ekisobozesa bizinensi okulinnyisa omutindo mu ngeri ey’amagezi. Mu nsi ya hardware ekulaakulana amangu, platform nga Mewayz egaba framework okussaamu obusobozi obupya nga tetuzzeemu kuzimba nnamuziga, oba mu mbeera yaffe, awatali kugezaako kuddamu kuyiiya processor. Engeri enzibu yatuyigiriza nti sipiidi ey’olubeerera si kuwangula buli lutalo olutonotono, wabula okukakasa nti eggye lyo lyonna lisobola okutambula mu ddoboozi limu.