Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Safe ConvolutionND #202

Merged
merged 22 commits into from
Feb 26, 2024
Merged

Safe ConvolutionND #202

merged 22 commits into from
Feb 26, 2024

Conversation

bloodre
Copy link
Contributor

@bloodre bloodre commented Feb 5, 2024

Summary

This PR introduces safe convolution operations for dimension N>=4, related safe APIs, tests, and deprecates Conv2d operations. Work was started after reading this issue content.

Content

  • Safe wrapper for cudnnSetFilterNdDescriptor
  • Safe wrapper for cudnnSetConvolutionNdDescriptor
  • Create of alias for Conv2dDescriptor, Conv2dForward, Conv2dBackwardData, Conv2dBackwardFilter: ConvDescriptor, ConvForward, ConvBackwardData, ConvBackwardFilter. The original struct names have also been deprecated. The reason is beside "2d" being included in the struct name, their methods apply to convolution operation in any dimension (pick_algorithm, get_workspace_size, launch).
  • Addition of tests: one for 3D convolution (using new APIs) and another for 1D conv (sample code with Conv2d operation)

Regarding 1D convolution

As far as I understood from the CUDNN API reference, the ConvolutionNd are not used for 1D convolution:

  • From cudnnSetTensorDescriptor: "Tensors are restricted to having at least 4 dimensions, and at most CUDNN_DIM_MAX dimensions (defined in cudnn.h). When working with lower dimensional data, it is recommended that the user create a 4D tensor, and set the size along unused dimensions to 1. ".
  • cudnnConvolutionForward will return CUDNN_STATUS_BAD_PARAM if the input tensor descriptor and weight tensor description dimensions mismatch. Using 3-dimensional filters is therefore not possible for convolutions.
    As shown in the test "test_conv1d" that I added in src/cudnn/safe/mod.rs, a workaround is therefore to define a 4D tensor with the last dimension equal to 1, and to use 2D convolution operation with a flattened kernel.

Note

  • It might be useful to create some utility function to generate the stride arrays of Nd tensors from the dimensions.

src/cudnn/safe/conv.rs Outdated Show resolved Hide resolved
@coreylowman
Copy link
Owner

Looks good, thanks for this!

@coreylowman coreylowman merged commit e533e10 into coreylowman:main Feb 26, 2024
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants