Skip to content

Kernel capture

In order to extract a kernel from a program, you have to:

  1. replace the desired Kokkos::parallel_for with krepe::parallel_for, this will allow to save the execution policy and the kernel's data
  2. execute with the libkrepe.so kokkos tool

Prepare your application

The library can be included in a CMake project:

find_package(KREPE REQUIRED)
target_link_libraries(my_program PRIVATE KREPE::extractor)

See Installation for the KREPE build instructions.

Instrument the kernel

Replace Kokkos::parallel_for with krepe::parallel_for for the kernel you want to capture. For example:

#include <Kokkos_Core.hpp>

#include <krepe/extractor.hpp>

int main(int argc, char* argv[]) {
  Kokkos::ScopeGuard kokkos_scope(argc, argv);

  const int N = 1024;
  Kokkos::View<int*> values("values", N);
  Kokkos::parallel_for(
      "init", values.size(), KOKKOS_LAMBDA(int i) { values(i) = i; });

  // Kokkos::parallel_for(
  //     "scale", N, KOKKOS_LAMBDA(int i) { values(i) *= 2; });
  krepe::parallel_for(
      "scale", N, KOKKOS_LAMBDA(int i) { values(i) *= 2; });
  Kokkos::fence();

  return 0;
}

Run the capture

The program has to be linked with KREPE::extractor, it then has to be executed with the following environment variables in order to extract the first invocation of the kernel named "scale"

KOKKOS_TOOLS_LIBS=/path/to/libkrepe.so \
KOKKOS_TOOLS_ARGS="--krepe-kernel-label=scale --krepe-kernel-invocation=1" \
./prog

Alternatively, use Kokkos command-line arguments:

./prog \
  --kokkos-tools-libs=/path/to/libkrepe.so \
  --kokkos-tools-args="--krepe-kernel-label=scale --krepe-kernel-invocation=1"

This generates one HDF5 file named krepe_scale_<kernel_id>.h5. See the HDF5 file layout for the naming rules, file structure and stored metadata. The file is written to the current working directory.

Parallel_for wrapper

If you are trying to extract a kernel used in a parallel_for wrapper provided by another library, you can use the krepe::replay_functor function to save the functor's data

my_funky_parallel_for("kernel", N,
                      krepe::replay_functor(
                          KOKKOS_LAMBDA(int i) { values(i) *= 2; }));

Here, my_funky_parallel_for represents your library's wrapper. Use this call in the initialized program above and select --krepe-kernel-label=kernel on the command line (assuming that the underlying Kokkos::parallel_for call uses the same label).

Note that the execution policy cannot be saved in this case.