8hp\ddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZejejeZejedgZd Zejd ZGd d eZd ZddddddZddddddZdZdZdZ GddeZ!Gdde Z"dS)N)Path)knobs) GPUTarget) GPUDriver)compile_module_from_src)TensorDescriptorincludec: ddl}|dkrdSddl ddlm}m}m}m m m}G fdd j } |||||||} dj }n#t$rYdSwxYw| g|_||_d dz} fd } ||| | r't#j | SdS) NrLinux)c_charc_intc_size_tc_void_pc_char_pPOINTERc eZdZdfdfgZdS)8_find_already_mmapped_dylib_on_linux..DlPhdrInfo dlpi_addr dlpi_nameN)__name__ __module__ __qualname___fields_)rrsb/var/www/html/services/rag-service/venv/lib/python3.11/site-packages/triton/backends/amd/driver.py DlPhdrInfors& ( # ( # rz libc.so.6ic |jj}ttj|}|jvr4||tt|dSdS)Nrr) contentsrrosfsdecodenamememmoveminlen)infosizedatarpctypeslib_namemax_path_lengths rcallbackz6_find_already_mmapped_dylib_on_linux..callback2saM+ Y'' ( ( qv   NN4CY,P,P Q Q Q1qr)platformsystemr*r r rrrr Structure CFUNCTYPECDLLdl_iterate_phdr Exceptionargtypesrestypecreate_string_bufferr r! string_at)r+r.r r rrr callback_tr3pathr-rrr*r,s` @@@@r$_find_already_mmapped_dylib_on_linuxr;sOOOG##t MMMKKKKKKKKKKKKKKKK        V%   !!%)<)O>OQXQXY_Q`Q`aaJ ++k22B tt!+H5O#OO  & &': ; ;Dzz(++T223{6++D11222 4sB B+*B+cddtjjx}rK|r!tj|r|Std|dt}|r6tj|r|Std|dg}tj tj td}tj|r|S| |ddl }|}|}|jr|g|z}|D]\}tj |dd}tj|r|cS| |]t jd }|rp|d D]Z} tj | } tj| r| cS| | [t'jd d g} fd | D} | D]:} tj| r| cS| | ;tj d}tj|r|S| |tdd|)Nzlibamdhip64.sozTRITON_LIBHIP_PATH 'z' does not point to a valid zmemory mapped 'z'' in process does not point to a valid librtorchLD_LIBRARY_PATH:z/sbin/ldconfigz-pcg|]C}|)|dDS))stripendswithsplit).0liner+s r z2_get_path_to_hip_runtime_dylib..usB ^ ^ ^djjll>S>ST\>]>] ^DJJLL  ^ ^ ^rz/opt/rocm/lib/zcannot locate z after attempted paths )ramd libhip_pathrDr r:exists RuntimeErrorr;joindirname__file__appendsitegetsitepackagesgetusersitepackagesENABLE_USER_SITEgetenvrE subprocess check_outputdecode splitlines)env_libhip_path mmapped_pathpaths local_librQ site_packages user_siter:env_ld_library_pathdflibslocsloccommon_install_pathr+s @r_get_path_to_hip_runtime_dylibrg@s.H )//k  # #H - - #"'..2Q2Q #" "i/ii_giijjj8AALn 7>>, ' '  l\llbjllmmm E RW__X66xHHI w~~i   LLKKK((**M((**I 4" m3 w||D'5(;; 7>>$   KKK T)$566$**3//  A Q))Aw~~a    LLOOOO  "$4d#; < < C C E ED _ ^ ^ ^):): ^ ^ ^D 7>>#   JJJ S',,'7BB w~~)**#"" LL$%%% PPPPP Q QQrc$eZdZfdZdZxZS)HIPUtilsct|ds-tt|||_|jS)Ninstance)hasattrsuperri__new__rk)cls __class__s rrnzHIPUtils.__new__s<sJ'' = 3//77<"""r)rrrrnr| __classcell__rps@rririsG ? ? ? ? ? ? ?rricN|ddkrdSddddddd d dd d d d d d d |S)Nr*hipDeviceptr_tint32_tint8_tint16_tint64_tuint32_tuint8_tuint16_tuint64_tdouble)i1i8i16i32i64u1u8u16u32u64fp16bf16fp32f32fp64)tys r ty_to_cpprs[ !u||    !  rrrr)rrrrr pack_fp16 pack_bf16 pack_fp32 pack_fp64 piiiKKOOOOcd}fdfdfddt||D}dfd|D}t|z}dt |}t t t|d}d t|D}t|d kr4d d d | Dznd}g}| D]b\}} | d kr | tvr&| t| d|;| t| d|cd |} g} | D]l\}} | d dkr| d|d+| tvr| d|dN| d kr| d|md| D} t} t tt|}d| D}| dd| dt| d krd | znddd |d|d|ddfd| Dd|d |d!d| d"d#d$| Dd%t| d krd d | zndd&}|S)'Nc6g}|D]}t|tr|dr|ddz}t jd|}|d|ztd|zD]}|dt|D]}|dt|D]}|d|||S) N tensordesc,rztensordesc<([^[>]*)rrr) isinstancestr startswithcountrematchgrouprPrange) signatureoutputsigndimdtype_s r_expand_signaturez(make_launcher.._expand_signatures* # #C#s## #|(D(D #yy~~)!6<<BBDD cEk***q4x))AMM%(((( t))AMM%((((t))AMM%(((() c"""" rcxt|tr#dt|S|S)Nr)rtuplerMmap)r_serialize_signatures rrz+make_launcher.._serialize_signatures7 c5 ! ! <88C 4c::;; ; rct|tr)dt|}d|dS|ddkrdS|dkrdSt |S)Nr[]rrz PyObject* constexprrrrMrr)rval_extracted_types rrz&make_launcher.._extracted_typesl b%  ((33344Cs:::  a5C<<;   ;}}rc t|tr)dt|}d|dS|ddkrdS|dkrdSdd d d d d ddddd t |S)N()rrOrralbhiLBHIK) rlongrrrrrrrrr)rr format_ofs rrz make_launcher..format_ofs b%  ''#i,,--Cs:::  a5C<<3   3    B--  rci|]\}}|| Srr)rFidxss r z!make_launcher..sWWWFCaWWWrrc&g|] }|Srr)rFrrs rrHz!make_launcher..s!FFFR99R==FFFrrci|]\}}|| Srr)rFrrs rrz!make_launcher..s777$!QA777rr, c3&K|] \}}d|V dS)z&_argNrrFrrs r z make_launcher..s, L LB L L L L L Lrrz argrptr_infoz.dev_ptr_arg_storagec ng|]2\}}|tvt|d|dt|d|d3S) _argz _storage = z(_argz);)FLOAT_STORAGE_TYPEFLOAT_PACK_FUNCTIONrs rrHz!make_launcher..s_ Ar # # # b !YYYY6I"6MYYTUYYY # # #rc*g|]\}}|dk d|S)rz&argrrs rrHz!make_launcher..&s, M M MUQ2;L;LjQjj;L;L;Lrz&global_scratcha; #define __HIP_PLATFORM_AMD__ #include #include #include #include #include // The list of paths to search for the HIP runtime library. The caller Python // code should substitute the search path placeholder. static const char *hipLibSearchPaths[] = {"aj"}; // The list of HIP dynamic library symbols and their signature we are interested // in this file. #define HIP_SYMBOL_LIST(FOR_EACH_ERR_FN, FOR_EACH_STR_FN) \ FOR_EACH_STR_FN(hipGetErrorString, hipError_t hipError) \ FOR_EACH_ERR_FN(hipModuleLaunchKernel, hipFunction_t f, \ unsigned int gridDimX, unsigned int gridDimY, \ unsigned int gridDimZ, unsigned int blockDimX, \ unsigned int blockDimY, unsigned int blockDimZ, \ unsigned int sharedMemBytes, hipStream_t stream, \ void **kernelParams, void **extra) \ FOR_EACH_ERR_FN(hipModuleLaunchCooperativeKernel, hipFunction_t f, \ unsigned int gridDimX, unsigned int gridDimY, \ unsigned int gridDimZ, unsigned int blockDimX, \ unsigned int blockDimY, unsigned int blockDimZ, \ unsigned int sharedMemBytes, hipStream_t stream, \ void **kernelParams, void **extra) \ FOR_EACH_ERR_FN(hipPointerGetAttribute, void *data, \ hipPointer_attribute attribute, hipDeviceptr_t ptr) // The HIP symbol table for holding resolved dynamic library symbols. struct HIPSymbolTable { #define DEFINE_EACH_ERR_FIELD(hipSymbolName, ...) \ hipError_t (*hipSymbolName)(__VA_ARGS__); #define DEFINE_EACH_STR_FIELD(hipSymbolName, ...) \ const char *(*hipSymbolName)(__VA_ARGS__); HIP_SYMBOL_LIST(DEFINE_EACH_ERR_FIELD, DEFINE_EACH_STR_FIELD) }; static struct HIPSymbolTable hipSymbolTable; bool initSymbolTable() { // Use the HIP runtime library loaded into the existing process if it exits. void *lib = dlopen("libamdhip64.so", RTLD_NOLOAD); // Otherwise, go through the list of search paths to dlopen the first HIP // driver library. if (!lib) { int n = sizeof(hipLibSearchPaths) / sizeof(hipLibSearchPaths[0]); for (int i = 0; i < n; ++i) { void *handle = dlopen(hipLibSearchPaths[i], RTLD_LAZY | RTLD_LOCAL); if (handle) { lib = handle; } } } if (!lib) { PyErr_SetString(PyExc_RuntimeError, "cannot open libamdhip64.so"); return false; } // Resolve all symbols we are interested in. dlerror(); // Clear existing errors const char *error = NULL; #define QUERY_EACH_FN(hipSymbolName, ...) \ *(void **)&hipSymbolTable.hipSymbolName = dlsym(lib, #hipSymbolName); \ error = dlerror(); \ if (error) { \ PyErr_SetString(PyExc_RuntimeError, \ "cannot query " #hipSymbolName " from libamdhip64.so"); \ dlclose(lib); \ return false; \ } HIP_SYMBOL_LIST(QUERY_EACH_FN, QUERY_EACH_FN) return true; } static inline void gpuAssert(hipError_t code, const char *file, int line) { if (code != HIP_SUCCESS) { const char* prefix = "Triton Error [HIP]: "; const char* str = hipSymbolTable.hipGetErrorString(code); char err[1024] = {0}; snprintf(err, 1024, "%s Code: %d, Messsage: %s", prefix, code, str ); PyErr_SetString(PyExc_RuntimeError, err); } } #define HIP_CHECK(ans) { gpuAssert((ans), __FILE__, __LINE__); } static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, hipStream_t stream, hipFunction_t functionz>) { hipDeviceptr_t global_scratch = 0; void *params[] = { z }; if (gridX*gridY*gridZ > 0 && launch_cooperative_grid) { HIP_CHECK(hipSymbolTable.hipModuleLaunchCooperativeKernel(function, gridX, gridY, gridZ, z*num_warps, 1, 1, shared_memory, stream, params, 0)); return; } if (gridX*gridY*gridZ > 0) { HIP_CHECK(hipSymbolTable.hipModuleLaunchKernel(function, gridX, gridY, gridZ, a *num_warps, 1, 1, shared_memory, stream, params, 0)); } } typedef struct _DevicePtrInfo { hipDeviceptr_t dev_ptr; bool valid; } DevicePtrInfo; static inline DevicePtrInfo getPointer(PyObject *obj, int idx) { DevicePtrInfo ptr_info; ptr_info.dev_ptr = 0; ptr_info.valid = true; if (PyLong_Check(obj)) { ptr_info.dev_ptr = (hipDeviceptr_t)PyLong_AsUnsignedLongLong(obj); return ptr_info; } if (obj == Py_None) { // valid nullptr return ptr_info; } PyObject *ptr = PyObject_GetAttrString(obj, "data_ptr"); if(ptr){ PyObject *empty_tuple = PyTuple_New(0); PyObject *ret = PyObject_Call(ptr, empty_tuple, NULL); Py_DECREF(empty_tuple); Py_DECREF(ptr); if (!PyLong_Check(ret)) { PyErr_SetString(PyExc_TypeError, "data_ptr method of Pointer object must return 64-bit int"); ptr_info.valid = false; return ptr_info; } ptr_info.dev_ptr = (hipDeviceptr_t)PyLong_AsUnsignedLongLong(ret); if(!ptr_info.dev_ptr) return ptr_info; uint64_t dev_ptr; hipError_t status = hipSymbolTable.hipPointerGetAttribute(&dev_ptr, HIP_POINTER_ATTRIBUTE_DEVICE_POINTER, ptr_info.dev_ptr); if (status == hipErrorInvalidValue) { PyErr_Format(PyExc_ValueError, "Pointer argument (at %d) cannot be accessed from Triton (cpu tensor?)", idx); ptr_info.valid = false; } ptr_info.dev_ptr = (hipDeviceptr_t)dev_ptr; Py_DECREF(ret); return ptr_info; } PyErr_SetString(PyExc_TypeError, "Pointer argument must be either uint64 or have data_ptr method"); return ptr_info; } static uint16_t pack_fp16(double f) { uint16_t result; // from https://github.com/python/pythoncapi-compat #if 0x030600B1 <= PY_VERSION_HEX && PY_VERSION_HEX <= 0x030B00A1 && !defined(PYPY_VERSION) _PyFloat_Pack2(f, (unsigned char*)&result, 1); #else PyFloat_Pack2(f, (unsigned char*)&result, 1); #endif return result; } static uint16_t pack_bf16(double f) { float f32 = (float)f; uint32_t u32 = *(uint32_t*)&f32; return (uint16_t)(u32 >> 16); } static uint32_t pack_fp32(double f) { float f32 = (float)f; return *(uint32_t*)&f32; } static uint64_t pack_fp64(double f) { return *(uint64_t*)&f; } static PyObject* launch(PyObject* self, PyObject* args) { int gridX, gridY, gridZ; uint64_t _stream; uint64_t _function; int launch_cooperative_grid; PyObject *launch_enter_hook = NULL; PyObject *launch_exit_hook = NULL; PyObject *kernel_metadata = NULL; PyObject *launch_metadata = NULL;  c8g|]\}}|d|dS)r; r)rFrrrs rrHz!make_launcher..s8 O O OEAr## / /! / / / O O Orz if(!PyArg_ParseTuple(args, "a", &launch_cooperative_grid, &gridX, &gridY, &gridZ, &_stream, &_function, &kernel_metadata, &launch_metadata, &launch_enter_hook, &launch_exit_hook z)) { return NULL; } a8 // extract kernel metadata int num_warps, num_ctas, shared_memory, clusterDimX, clusterDimY, clusterDimZ; if (!PyArg_ParseTuple(kernel_metadata, "iiiiii", &num_warps, &num_ctas, &shared_memory, &clusterDimX, &clusterDimY, &clusterDimZ)) { return NULL; } // extract launch metadata if (launch_enter_hook != Py_None){ PyObject* args = Py_BuildValue("(O)", launch_metadata); PyObject* ret = PyObject_CallObject(launch_enter_hook, args); Py_DECREF(args); if (!ret) return NULL; Py_DECREF(ret); } // raise exception asap rc Ng|]"\}}|ddkrd|d|d|d|d nd#S)rrzDevicePtrInfo ptr_infoz = getPointer(_argrz); if (!ptr_infoz.valid) return NULL;rrrs rrHz!make_launcher..s~dddINIJLNoqrsotx{o{o{kqkkAkkkkTUkkkkBDdddrz; _launch(gridX, gridY, gridZ, num_warps, num_ctas, launch_cooperative_grid, clusterDimX, clusterDimY, clusterDimZ, shared_memory, (hipStream_t)_stream, (hipFunction_t)_functiona); if(launch_exit_hook != Py_None){ PyObject* args = Py_BuildValue("(O)", launch_metadata); PyObject* ret = PyObject_CallObject(launch_exit_hook, args); Py_DECREF(args); if (!ret) return NULL; Py_DECREF(ret); } if(PyErr_Occurred()) { return NULL; } // return None Py_INCREF(Py_None); return Py_None; } static PyMethodDef ModuleMethods[] = { {"launch", launch, METH_VARARGS, "Entry point for all kernels with this signature"}, {NULL, NULL, 0, NULL} // sentinel }; static struct PyModuleDef ModuleDef = { PyModuleDef_HEAD_INIT, "__triton_launcher", NULL, //documentation -1, //size ModuleMethods }; PyMODINIT_FUNC PyInit___triton_launcher(void) { if (!initSymbolTable()) { return NULL; } PyObject *m = PyModule_Create(&ModuleDef); if(m == NULL) { return NULL; } PyModule_AddFunctions(m, ModuleMethods); return m; } ) enumeratevaluesrM_BASE_ARGS_FORMATrlistfilterboolrEr%itemsrrPrrgr) constantsr warp_sizer args_formatformat args_list arg_decl_listrr arg_declsinternal_args_listfloat_storage_declsrJparamsrtrrrs @@@r make_launcherrs4 *XWi0A0A)BRBRBTBT0U0U&V&VWWWI''FFFF93C3C3E3EFFFGGK  ,F193C3C3E3EFFGGIVD)//#"6"67788I77)I"6"6777IPST]P^P^abPbPbtyy L L)//:K:K L L LLLLLhjIM""<<2     # # #  $6r$:!C!C!C!C D D D D  IbMM!:!:q!:!: ; ; ; ; -((I""222 a5C<<  % %&<&<&<&< = = = = % % %  % %&8Q&8&8&8 9 9 9 9 ;    % %jQjj 1 1 1__&& 122K%I'' ( (F M Mioo&7&7 M M MF MM#$$$C .9C C ~uxyBuCuCFGuGuGaehqaqaqMOC C Byy((CC C F_hGC C NT]OC C x88 O O O OY__=N=N O O OPPyC C z!'{C C @S\AC C H88   IC C n99ddR[RaRaRcRcdddeeoC C p[^_q[r[ruv[v[vswz~zCzCDVzWzWsWsW|~qC C C CH Jrcfd}|S)zN Replace all tensor descriptors with the base ptr, shape, and strides cZ|dtt}|ttd}g}|D]d}t|tr8||jg|j|j|j|jO||eg||RSN) r%rrrextendbaseshapestridesrP)args meta_argsraw_kernel_args final_argsarglaunchers rinnerz,wrap_handle_tensor_descriptor..inner4s0#/0001 s#455667 " ' 'C#/00 '!!38"`ci"`#+"` "`TWT_"`aaaa!!#&&&&x00Z0000rr)rrs` rwrap_handle_tensor_descriptorr/s# 11111" LrceZdZdZdZdS) HIPLauncherctdrjn t}fdfd|D}djD}t |||jtdt}td| D}|rt|j n|j |_ |j |_ dS)Nrcrt|tr jj|fn|Sr)rrfn arg_namesindex)xrts rz&HIPLauncher.__init__..Ls2Z3=O=OVSV-33A6699UVrc.i|]\}}||Srr)rFrvaluearg_idxs rrz(HIPLauncher.__init__..Ms'MMMZS%WWS\\5MMMrci|]\}}|| Srr)rFrrs rrz(HIPLauncher.__init__..NsHHHJCS%HHHr__triton_launcherrsc3jK|].}t|to|dV/dS)rN)rrr)rFrs rrz'HIPLauncher.__init__..Qs>!v!v\_*S#"6"6"W3>>,;W;W!v!v!v!v!v!vr)rlrdictrrrrrruanyrrlaunchlaunch_cooperative_grid)rzrtmetadatarrr{has_tensor_desc_argrs ` @rr|zHIPLauncher.__init__Js%,S+%>%>JCMMDFF VVVVMMMM9??;L;LMMM HH#-2E2E2G2GHHH Iy(2DEE%#4GVbccc!!v!vclcscscucu!v!v!vvvCVf3CJ???\_\f '/'G$$$rc,|j|jg|RdSr)rr)rzrs r__call__zHIPLauncher.__call__Vs% D084888888rN)rrrr|rrrrr r Hs5 H H H99999rr cXeZdZfdZdZedZdZdZdZ dZ dZ xZ S) HIPDriverctt|_t|_dSr)rmr|riutilsr  launcher_cls)rzrps rr|zHIPDriver.__init__\s2 ZZ 'rcddl}|jS)Nr)r>cudarzr>s rget_device_interfacezHIPDriver.get_device_interfaceas zrc| ddl}|jo |jjduS#t $rYdSwxYw)NrF)r>r% is_availableversionhip ImportError)r>s r is_activezHIPDriver.is_activeesU  LLL:**,,P%-2C42O P   55 s *- ;;c|}|j|}tjjp|d}|d}t d|dd|S)NarchwarpSizer+r@r)get_current_devicer"ryrruntime override_archrrE)rzdevicedevice_propertiesr/rs rget_current_targetzHIPDriver.get_current_targetmsi((** J<>>rcZddl}|d|S)Nrr%)r>r4r1r&s rget_active_torch_devicez!HIPDriver.get_active_torch_devicets+ ||FD$;$;$=$=>>>rcddlm}|S)Nr)do_bench)triton.testingr:)rzr:s rget_benchmarkerzHIPDriver.get_benchmarkerys++++++rchddl}d}|t|dz|jdS)Nrir%)rr4)r>emptyint)rzr> cache_sizes rget_empty_cache_for_benchmarkz'HIPDriver.get_empty_cache_for_benchmark}s8 ' {{3zQ//uy{PPPrc.|dSr)zero_)rzcaches r clear_cachezHIPDriver.clear_caches r) rrrr|r' staticmethodr-r6r8r<rBrFr}r~s@rr r Zs((((( \?????? QQQrr )# functoolsr rVrpathlibrtritonrtriton.backends.compilerrtriton.backends.driverrtriton.runtime.buildrtriton.tools.tensor_descriptorrr:rNrealpathrOrMrur; lru_cachergobjectrirrrrrrr r rrrrRs  ......,,,,,,888888;;;;;; '//"'**844 5 5 Wi001 ---`@R@R@RF?????v???(   .            !lll^ 299999&999$+++++ +++++r