在测量执行时间之前，您可以将输入特征移动到模型设备上吗？

我进行了一些测试，推理速度结果对于编码来说可疑，但对于生成来说则不是。
一个是使用post方法：
CTranslate2/src/models/whisper.cc
第656行
| | return post( |
另一个是使用post_batch:
CTranslate2/src/models/whisper.cc
第667行
| | return post_batch( |
这里是Guillaume的PR,其中实现了encode方法。我不知道为什么Guillaume没有使用post_batch。也许encode方法只依赖于cuda代码，这在与批量相比应该对具有批次的Tensor效率更高的tensor上运行高级逻辑？
#1144
我可以盲目尝试使用post_batch。
@vince62s对此有什么想法吗？或者你能在ct2上联系到可以帮助的人吗？

在测量执行时间之前，您可以将输入特征移动到模型设备吗？
10个样本。
更改前

def get_ctranslate2_storage(segment: np.ndarray) -> ctranslate2.StorageView:
    segment = np.ascontiguousarray(segment)
    segment = ctranslate2.StorageView.from_array(segment)
    return segment

prepare features cost: 0.527921199798584
get_ctranslate2_storage cost 0.013983964920043945 seconds 
self.model.encode [10, 128, 3000]  cost 5.466844797134399 seconds 
encoder_output shape:[10, 1500, 1280]
encode cost: 5.481068134307861

更改后

def get_ctranslate2_storage(segment: np.ndarray) -> ctranslate2.StorageView:
    segment = np.ascontiguousarray(segment)
    segment = torch.from_numpy(segment).float().to("cuda:0")  #added line.
    segment = ctranslate2.StorageView.from_array(segment)
    return segment

prepare features cost: 0.47109270095825195
get_ctranslate2_storage cost 0.6899769306182861 seconds 
self.model.encode [10, 128, 3000]  cost 5.4121253490448 seconds 
encoder_output shape:[10, 1500, 1280]
encode cost: 6.102340459823608

结论：

1. 在get_ctranslate2_storage中将特征从cpu移动到cuda:0(模型所在位置),导致0.0139秒 -> 0.6899秒，编码5.4668秒 -> 5.4121秒。
2. 因此，问题仍然来自CT2编码，而不是在CPU和GPU之间复制特征。

这是一个更详细的使用whisper tiny的分析：

45.09%  77.56%  45.09% MultiHeadAttention      13.81ms
14.73%  14.73%  59.82% BiasAdd                 4.51ms
12.79%  27.28%  72.61% dot_product_attention   3.92ms
 7.45%   7.45%  80.06% SoftMax                 2.28ms
 7.04%   7.04%  87.10% MatMul                  2.16ms
 6.88%  21.61%  93.98% Gemm                    2.11ms
 2.84%   2.84%  96.83% Conv1D                  0.87ms
 0.79%   0.79%  97.62% Transpose               0.24ms
 0.64%   0.64%  98.26% LayerNorm               0.20ms
 0.50%   0.50%  98.76% Split                   0.15ms
 0.37%   0.37%  99.13% Add                     0.11ms
 0.27%  96.59%  99.40% TransformerEncoderLayer 0.08ms
 0.20%  99.90%  99.60% WhisperEncoder          0.06ms
 0.17%  21.78%  99.76% Dense                   0.05ms
 0.13%   0.13%  99.90% GELU                    0.04ms
0.164976 0.107653 0.0736297 ... -0.023933 -0.588876 -0.235716
Batch size: 1, Execution time: 0.035004863739013674 seconds, Execution time per batch: 0.035004863739013674 seconds

35.34%  50.61%  35.34% dot_product_attention   17.50ms
27.67%  91.61%  63.01% MultiHeadAttention      13.70ms
14.65%  15.88%  77.66% Gemm                    7.25ms
 7.84%   7.84%  85.51% SoftMax                 3.88ms
 7.42%   7.42%  92.93% MatMul                  3.67ms
 2.03%   2.03%  94.95% Conv1D                  1.00ms
 1.25%  97.56%  96.21% TransformerEncoderLayer 0.62ms
 1.22%   1.22%  97.43% BiasAdd                 0.61ms
 0.75%   0.75%  98.18% Transpose               0.37ms
 0.59%   0.59%  98.77% LayerNorm               0.29ms
 0.44%   0.44%  99.21% Split                   0.22ms
 0.38%   0.38%  99.59% Add                     0.19ms
 0.14%  99.95%  99.73% WhisperEncoder          0.07ms
 0.12%  15.99%  99.84% Dense                   0.06ms
 0.11%   0.11%  99.95% GELU                    0.06ms
0.164976 0.107653 0.0736297 ... -0.0239328 -0.588876 -0.235715
Batch size: 2, Execution time: 0.058016990661621096 seconds, Execution time per batch: 0.029008495330810548 seconds

43.45%  92.19%  43.45% MultiHeadAttention      40.96ms
18.09%  32.20%  61.55% dot_product_attention   17.05ms
 9.94%   9.94%  71.49% Transpose               9.37ms
 8.76%   9.83%  80.25% Gemm                    8.26ms
 7.78%   7.78%  88.03% SoftMax                 7.34ms
 6.33%   6.33%  94.36% MatMul                  5.96ms
 2.64%   2.64%  97.00% Conv1D                  2.49ms
 1.07%   1.07%  98.07% BiasAdd                 1.01ms
 0.56%  97.08%  98.63% TransformerEncoderLayer 0.53ms
 0.44%   0.44%  99.07% LayerNorm               0.41ms
 0.35%   0.35%  99.42% Add                     0.33ms
 0.35%   0.35%  99.77% Split                   0.33ms
 0.10%   0.10%  99.86% GELU                    0.09ms
 0.08%  99.99%  99.94% WhisperEncoder          0.07ms
 0.05%   9.88%  99.99% Dense                   0.04ms
0.164977 0.107653 0.07363 ... -0.0239328 -0.588876 -0.235715
Batch size: 4, Execution time: 0.09641846466064453 seconds, Execution time per batch: 0.024104616165161134 seconds


31.35%  86.33%  31.35% MultiHeadAttention      47.57ms
26.44%  44.62%  57.79% dot_product_attention   40.13ms
10.99%  10.99%  68.78% SoftMax                 16.68ms
 8.05%   9.28%  76.83% Gemm                    12.22ms
 7.19%   7.19%  84.02% MatMul                  10.91ms
 4.52%  97.61%  88.54% TransformerEncoderLayer 6.86ms
 2.79%   2.79%  91.33% Transpose               4.24ms
 2.39%   2.39%  93.72% Split                   3.63ms
 2.35%   2.35%  96.08% LayerNorm               3.57ms
 2.08%   2.08%  98.16% Conv1D                  3.16ms
 1.23%   1.23%  99.39% BiasAdd                 1.87ms
 0.38%   0.38%  99.77% Add                     0.57ms
 0.11%   0.11%  99.88% GELU                    0.17ms
 0.07%  99.98%  99.96% WhisperEncoder          0.11ms
 0.03%   9.31%  99.98% Dense                   0.04ms
0.164976 0.107653 0.0736298 ... -0.0239328 -0.588876 -0.235715
Batch size: 8, Execution time: 0.1561301727294922 seconds, Execution time per batch: 0.019516271591186525 seconds

32.22%  88.50%  32.22% MultiHeadAttention      104.86ms
27.55%  48.49%  59.77% dot_product_attention   89.65ms
11.62%  11.62%  71.39% MatMul                  37.82ms
 9.33%   9.33%  80.72% SoftMax                 30.36ms
 6.91%   8.00%  87.62% Gemm                    22.49ms
 4.16%  97.41%  91.79% TransformerEncoderLayer 13.55ms
 1.69%   1.69%  93.48% Transpose               5.51ms
 1.69%   1.69%  95.17% Conv1D                  5.50ms
 1.35%   1.35%  96.52% Split                   4.38ms
 1.23%   1.23%  97.75% LayerNorm               4.02ms
 1.09%   1.09%  98.84% BiasAdd                 3.56ms
 0.72%  99.99%  99.56% WhisperEncoder          2.33ms
 0.32%   0.32%  99.89% Add                     1.05ms
 0.09%   0.09%  99.98% GELU                    0.30ms
 0.01%   8.02%  99.99% Dense                   0.04ms
0.164977 0.107653 0.07363 ... -0.023933 -0.588876 -0.235715
Batch size: 16, Execution time: 0.3303467102050781 seconds, Execution time per batch: 0.020646669387817382 seconds

以下是CT2生成函数的性能分析，它收集了编码和解码。似乎在beam_search中的解码通过批处理非常高效，并代表了主要耗时任务。
另一方面，WhisperEncoder中的编码并没有真正从编码中受益。可能CT2本身没有问题，而是架构/计算中的瓶颈使得批处理不能使batch_size倍速加快。编码器批处理仍然有小幅改进，只是与batch_size不成比例。
此外，副本池内部进行了一些优化，如果理解正确的话，它正在进行无声批处理，因此无声批处理可以解释为什么ct2 whisper很快，为什么手动批处理没有带来太大的改进。
我会在这里停止，因为我找不到任何明显的问题，请随意继续调查。

你好，你在使用torch.profiler或python cProfile来获取上述分析结果吗？

我正在使用ct2分析器。

你好！
你对这个有什么消息吗？我正在使用更快的whisper构建一些东西，我刚刚开始注意到批处理中元素数量的相对时间增加。

你好@funboarder13920, @dyyzhmm,
我想知道如何使用ct2分析，因为关于这方面的文档不多。
我非常感谢在这方面的帮助或指导。
你能指出调用分析的正确方法吗？或者提供一个示例，这将非常有帮助。
提前感谢。