Dim3 grid calculation code#

In StyleGAN2, up/downsampling operations use a separable 2D Gaussian kernel ( ). downsampling (by dropping zeros after each pixel).FIR filter (by convolution with a kernel).padding (by adding zeros on each side of the image).upsampling (by adding zeros after each pixel).image in StyleGAN2) goes through the following steps: The upfirdn2d is a very common signal processing operation where a signal (i.e. The former, on the other hand, is used for up/downsampling (by replacing the bilinear up/downsampling in the original StyleGAN paper). The latter is very simple as it is just a leaky ReLU that comes with a bias term. The StyleGAN2 team implemented two custom kernels: upfirdn2d and used_bias_act. Hence, I took the time and effort to investigate its inner workings, simplify it and summarize my learnings here.

Dim3 grid calculation code#

Sadly, the source code was not properly commented nor systematically explained anywhere. I wanted to understand the implementation of the CUDA kernels and the design considerations that went into the process. In configuration E at $1024^2$ resolution, our optimizations improved the overall training time by about 30% and memory footprint by about 20%. We implemented filtered up/downsampling as a single fused operation, and bias and activation as another one. This motivated us to optimize these operations using hand-written CUDA kernels. In the StyleGAN2 paper, the team mentioned that they implemented custom CUDA kernels to speed up the training time. I found the latter more stable during training and much simpler (such that I can incorporate it in my project repo more easily). I experimented with both NVIDIA’s official implementation and rosinality’s implementation. But I don't know whether that is reasonable for GPU computing.During my final year project in university, I needed to train a StyleGAN2 model for X-ray image synthesis. There are ways to do calculations in DP without DP units. You may want to test your code on another GPU or use single precision and check if it runs noticeably faster. (Inaccurate time measuring may be a problem, too.) On my GPU without doing any calculations I still get about 20% of the runtime with calculations (without optimization flags). GPUs reach maximal performance as the problem size grows. Even for your GPU, the kernel runtime for peak performance would be only about 0.17ms (or 0.23ms). Additionally every thread has only 10 double precision operations (14 if a is calculated twice for a*a) which results in a total of 2.5 MFLOP (or 3.5 MFLOP). You are launching kernels with a grid size of only 16x16 blocks or ~250k threads. WorkloadĪnother problem is the small problem size. This results in a peak performance of about 15 GFLOP/s only (~0.95GHz). Two suggestions why your runtime is so long: HardwareĪs you want to do the calculations with double precision you should look out for hardware that provides many more double precision units. The GPU device I am using is NVidia Quadro K2000, and my CUDA version is 7.5. So please consider that the data have sorted to optimize the memory access.Īll the calculations must be done with double precision to decrease the round off errors. I have sorted the data in the original code before using them in kernel which I think it has a positive effect on future memory accesses. Std::cout << 'KERNEL TIME = ' << distanceCheck << ' milliseconds' << std::endl Thrust::raw_pointer_cast(&particles2),ĬudaEventElapsedTime(&dummymili, start, stop) GridSize.y = (particles2.size() + blockSize.y - 1) / blockSize.y GridSize.x = (particles1.size() + blockSize.x - 1) / blockSize.x Thrust::device_vector distance(particles1.size()*particles2.size(), true) ĭim3 blockSize(32,32) // also tested for blockSize(16,16) Thrust::device_vector d_xPos(h_xPos), d_yPos(h_yPos), d_zPos(h_zPos), d_h(h_h) The kernel calculates the distance between several spatial points and based on whether they are neighbors or not, it fills a boolean vector.įor (int i = 0 i h_xPos(num), h_yPos(num), h_zPos(num), h_h(num,0.001) I have a kernel function which is consuming more than 70% of execution time. I am developing a CUDA program and I want to enhance my performance.