我正在尝试并行化这个numba jitted函数。我最初认为这是微不足道的,因为这是一个令人尴尬的并行问题,但它产生了意想不到的结果(顺序和并行“实现”的不同输出)。你知道那里会发生什么吗?是否有办法让这一切顺利进行?
参见以下可重现示例:
import numba
import numpy as np
from sklearn.datasets import make_regression
@numba.jit(nopython=True)
def nanlstsq(X, y):
"""Return the least-squares solution to a linear matrix equation
Analog to ``numpy.linalg.lstsq`` for dependant variable containing ``Nan``
Args:
X ((M, N) np.ndarray): Matrix of independant variables
y ({(M,), (M, K)} np.ndarray): Matrix of dependant variables
Returns:
np.ndarray: Least-squares solution, ignoring ``Nan``
"""
beta = np.zeros((X.shape[1], y.shape[1]), dtype=np.float64)
for idx in range(y.shape[1]):
# subset y and X
isna = np.isnan(y[:,idx])
X_sub = X[~isna]
y_sub = y[~isna,idx]
# Compute beta on data subset
XTX = np.linalg.inv(np.dot(X_sub.T, X_sub))
XTY = np.dot(X_sub.T, y_sub)
beta[:,idx] = np.dot(XTX, XTY)
return beta
@numba.jit(nopython=True, parallel=True)
def nanlstsq_parallel(X, y):
beta = np.zeros((X.shape[1], y.shape[1]), dtype=np.float64)
for idx in numba.prange(y.shape[1]):
# subset y and X
isna = np.isnan(y[:,idx])
X_sub = X[~isna]
y_sub = y[~isna,idx]
# Compute beta on data subset
XTX = np.linalg.inv(np.dot(X_sub.T, X_sub))
XTY = np.dot(X_sub.T, y_sub)
beta[:,idx] = np.dot(XTX, XTY)
return beta
# Generate random data
n_targets = 10000
n_features = 3
X, y = make_regression(n_samples=200, n_features=n_features,
n_targets=n_targets)
# Add random nan to y array
y.ravel()[np.random.choice(y.size, 5*n_targets, replace=False)] = np.nan
# Run the regression
beta = nanlstsq(X, y)
beta_parallel = nanlstsq_parallel(X, y)
np.testing.assert_allclose(beta, beta_parallel)
字符串
1条答案
按热度按时间7kqas0il1#
不确定是什么导致了顺序和并行版本之间的差异,但不建议计算
X_sub.T @ X_sub
的逆来求解矩阵方程-don't invert that matrix。如果你仍然想计算法方程,那么你应该使用numpy.linalg.solve或者不计算法方程,你可以使用numpy.linalg.lstsq,这两个都与numba兼容,并通过你的assert_allclose
测试:字符串