python-3.x 异常：HttpResponseError：(BadRequest)未找到实体[pipeline_name]

6jygbczu  于 2023-04-22  发布在  Python
关注(0)|答案(1)|浏览(75)

我在Azure Data Factory中创建了一个管道，它接受Avro文件并从中创建SQL表。我已经在ADF中测试了管道，它工作正常。现在我需要从Azure函数触发此管道：为此，我尝试在函数中使用以下代码创建管道的运行：

import logging
from azure.identity import DefaultAzureCredential
from azure.mgmt.datafactory import DataFactoryManagementClient
import azure.functions as func

def main(req: func.HttpRequest) -> func.HttpResponse:
    from .config import suscription_id, resource_group, factory_name, pipeline_name
    client = DataFactoryManagementClient(
        credential=DefaultAzureCredential(),
        subscription_id=suscription_id,
    )
    logging.info('==== ADF client created ====')

    response = client.pipelines.create_run(
        resource_group_name=resource_group,
        factory_name=factory_name,
        pipeline_name=pipeline_name,
    )
    logging.info(response)
    return func.HttpResponse(f"Pipeline {pipeline_name} started")

但由于某种原因，我收到一条错误消息，说无法找到资源，引用管道。我已经验证了所有名称，一切似乎都是正确的。你有什么建议来调试这个吗？

[2023-04-13T04:34:00.591Z] Executed 'Functions.restoreTable' (Failed, Id=c00a9f36-695d-4806-bb6d-880ac88f1d95, Duration=7398ms)
[2023-04-13T04:34:00.591Z] System.Private.CoreLib: Exception while executing function: Functions.restoreTable. System.Private.CoreLib: Result: Failure
[2023-04-13T04:34:00.591Z] Exception: HttpResponseError: (BadRequest) Entity Restore_backup not found
[2023-04-13T04:34:00.591Z] Code: BadRequest
[2023-04-13T04:34:00.591Z] Message: Entity Restore_backup not found
[2023-04-13T04:34:00.591Z] Target: pipeline/Restore_backup/runid/65350dbc-d9b4-11ed-8e61-acde48001122
[2023-04-13T04:34:00.591Z] Stack:   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5095/workers/python/3.9/OSX/X64/azure_functions_worker/dispatcher.py", line 452, in _handle__invocation_request
[2023-04-13T04:34:00.591Z]     call_result = await self._loop.run_in_executor(
[2023-04-13T04:34:00.591Z]   File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
[2023-04-13T04:34:00.591Z]     result = self.fn(*self.args, **self.kwargs)
[2023-04-13T04:34:00.591Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5095/workers/python/3.9/OSX/X64/azure_functions_worker/dispatcher.py", line 718, in _run_sync_func
[2023-04-13T04:34:00.591Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2023-04-13T04:34:00.591Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5095/workers/python/3.9/OSX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2023-04-13T04:34:00.591Z]     result = function(**args)
[2023-04-13T04:34:00.591Z]   File "/Users/jd/Documents/globant-api/restoreTable/__init__.py", line 14, in main
[2023-04-13T04:34:00.591Z]     response = client.pipelines.create_run(
[2023-04-13T04:34:00.592Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
[2023-04-13T04:34:00.592Z]     return func(*args, **kwargs)
[2023-04-13T04:34:00.592Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/mgmt/datafactory/operations/_pipelines_operations.py", line 937, in create_run
[2023-04-13T04:34:00.592Z]     raise HttpResponseError(response=response, error_format=ARMErrorFormat)

更新

我相信我的应用尝试使用的凭据存在冲突。我已经将我的Azure帐户添加到Vscode登录并安装了Azure CLI以登录其中。但是，一旦我尝试使用Postman访问端点，就会打开一个对话框，要求允许在Keychain中使用Azure凭据：

当我拒绝请求时，我收到以下消息

2023-04-13T18:18:56.002Z] DefaultAzureCredential failed to retrieve a token from the included credentials.
[2023-04-13T18:18:56.002Z] Attempted credentials:
[2023-04-13T18:18:56.002Z]      EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
[2023-04-13T18:18:56.002Z] Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.
[2023-04-13T18:18:56.003Z]      ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
[2023-04-13T18:18:56.003Z]      SharedTokenCacheCredential: Authentication failed: 
[2023-04-13T18:18:56.003Z] To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[2023-04-13T18:18:56.128Z] Executed 'Functions.restoreTable' (Failed, Id=3c3d2f26-1d37-4b0f-a94e-b3d4d4978c08, Duration=21193ms)
[2023-04-13T18:18:56.128Z] System.Private.CoreLib: Exception while executing function: Functions.restoreTable. System.Private.CoreLib: Result: Failure
[2023-04-13T18:18:56.128Z] Exception: ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
[2023-04-13T18:18:56.128Z] Attempted credentials:
[2023-04-13T18:18:56.128Z]      EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
[2023-04-13T18:18:56.128Z] Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.
[2023-04-13T18:18:56.128Z]      ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
[2023-04-13T18:18:56.128Z]      SharedTokenCacheCredential: Authentication failed: 
[2023-04-13T18:18:56.128Z] To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[2023-04-13T18:18:56.128Z] Stack:   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5095/workers/python/3.9/OSX/X64/azure_functions_worker/dispatcher.py", line 452, in _handle__invocation_request
[2023-04-13T18:18:56.128Z]     call_result = await self._loop.run_in_executor(
[2023-04-13T18:18:56.128Z]   File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
[2023-04-13T18:18:56.128Z]     result = self.fn(*self.args, **self.kwargs)
[2023-04-13T18:18:56.128Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5095/workers/python/3.9/OSX/X64/azure_functions_worker/dispatcher.py", line 718, in _run_sync_func
[2023-04-13T18:18:56.128Z]     return ExtensionManager.get_sync_invocation_wrapper(context,
[2023-04-13T18:18:56.128Z]   File "/usr/local/Cellar/azure-functions-core-tools@4/4.0.5095/workers/python/3.9/OSX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
[2023-04-13T18:18:56.128Z]     result = function(**args)
[2023-04-13T18:18:56.128Z]   File "/Users/jd/Documents/globant-api/restoreTable/__init__.py", line 14, in main
[2023-04-13T18:18:56.128Z]     response = client.pipelines.create_run(
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
[2023-04-13T18:18:56.129Z]     return func(*args, **kwargs)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/mgmt/datafactory/operations/_pipelines_operations.py", line 929, in create_run
[2023-04-13T18:18:56.129Z]     pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 202, in run
[2023-04-13T18:18:56.129Z]     return first_node.send(pipeline_request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 70, in send
[2023-04-13T18:18:56.129Z]     response = self.next.send(request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 70, in send
[2023-04-13T18:18:56.129Z]     response = self.next.send(request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 70, in send
[2023-04-13T18:18:56.129Z]     response = self.next.send(request)
[2023-04-13T18:18:56.129Z]   [Previous line repeated 2 more times]
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
[2023-04-13T18:18:56.129Z]     response = self.next.send(request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect.py", line 156, in send
[2023-04-13T18:18:56.129Z]     response = self.next.send(request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry.py", line 448, in send
[2023-04-13T18:18:56.129Z]     response = self.next.send(request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 108, in send
[2023-04-13T18:18:56.129Z]     self.on_request(request)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 87, in on_request
[2023-04-13T18:18:56.129Z]     self._token = self._credential.get_token(*self._scopes)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/identity/_credentials/default.py", line 168, in get_token
[2023-04-13T18:18:56.129Z]     return super(DefaultAzureCredential, self).get_token(*scopes, **kwargs)
[2023-04-13T18:18:56.129Z]   File "/Users/jd/Documents/globant-api/.venv/lib/python3.9/site-packages/azure/identity/_credentials/chained.py", line 101, in get_token
[2023-04-13T18:18:56.129Z]     raise ClientAuthenticationError(message=message)
[2023-04-13T18:18:56.129Z] .

当我接受请求时，我会收到一条初始消息，说我找不到资源（管道），如果这是使用另一个没有我的管道的帐户的凭据，这是有道理的。我如何强制代码使用Vscode或CLI凭据？

python-3.x

1条答案

按热度按时间
nvbavucw

nvbavucw1#

我使用下面的代码来触发我的ADF管道，该管道获取avro文件并从中创建SQL表：

我的init.py文件：-

import logging
import os
import json
from azure.identity import DefaultAzureCredential
from azure.mgmt.datafactory import DataFactoryManagementClient
import azure.functions as func

def main(req: func.HttpRequest, res: func.Out[func.HttpResponse]) -> None:
    try:
        # Load configuration values from config.json
        with open(os.path.abspath(os.path.join(os.path.dirname(__file__), 'config.json')), 'r') as f:
            config = json.load(f)

        # Create Data Factory Management Client
        client = DataFactoryManagementClient(
            credential=DefaultAzureCredential(),
            subscription_id=config['subscription_id']
        )
        logging.info('ADF client created')

        # Trigger Pipeline
        run_response = client.pipelines.create_run(
            resource_group_name=config['resource_group_name'],
            factory_name=config['factory_name'],
            pipeline_name=config['pipeline_name']
        )

        # Return response
        res.set(func.HttpResponse(f"Pipeline run_id {run_response.run_id} created"))
        
    except Exception as e:
        logging.error(str(e))
        res.set(func.HttpResponse(f"An error occurred: {str(e)}", status_code=500))

我的config.json文件：-

{

"subscription_id": "<subscription-id>",

"resource_group_name": "<resource-group-name>",

"factory_name": "<adf-name>",

"pipeline_name": "pipeline6"

}

我的函数.json

{

"bindings": [

{

"name": "req",

"type": "httpTrigger",

"direction": "in",

"authLevel": "anonymous",

"methods": [

"post"

]

},

{

"type": "http",

"direction": "out",

"name": "res"

}

],

"scriptFile": "__init__.py",

"entryPoint": "main"

}

我的需求.txt：-

azure-functions

azure-mgmt-datafactory

azure-identity

我使用HTTP post请求从函数目录中找到config.json，其中包含ADF和管道详细信息并触发管道，参考如下：

输出：-

Postman中的发布请求：-

入口：-
Avro文件到SQL管道：-

通过Azure函数成功触发管道：-

Avro文件被添加到SQL表中，如下所示：-

赞(0）分享  回复(0）举报  2023-04-22
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com