Meltano:(tag and tagret:postgresql)临时文件大小超过temp_file_limit

quhf5bfb  于 2023-06-22  发布在  PostgreSQL
关注(0)|答案(1)|浏览(157)

我在运行Meltano时遇到了这个问题,使用标签和目标“postgresql”作为源数据库,而没有通过点击“postgresql”来更改temp_file_limit参数。我该怎么弥补?
psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (13411002kB)
Meltanomeltano.yml设置文件如下所示:

default_environment: dev
project_id: fake-id
plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
  - name: tap-google-analytics
    variant: meltanolabs
    pip_url: pip install git+https://github.com/MeltanoLabs/tap-google-analytics.git
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      schema_mapping:
        schema_1:
          target_schema: replica_schema_1  
      default_target_schema: public
    
  orchestrators:
  - name: airflow
    pip_url: psycopg2 apache-airflow==2.1.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt
  files:
  - name: airflow
    pip_url: git+https://gitlab.com/meltano/files-airflow.git

schedules:
- name: schedule-name
  extractor: tap-postgres
  loader: target-postgres
  transform: skip
  interval: '*/30 * * * *'

environments:

- name: dev
  config:
    plugins:
      extractors:
      - name: tap-postgres
        config:
          dbname: $TAP_POSTGRES_DBNAME
          user: $TAP_POSTGRES_USER
        
        select:
        - schema_name-table_name_1
        - schema_name-table_name_2

      loaders:
      - name: target-postgres
        config:
          user: $TARGET_POSTGRES_USER
          dbname: $TARGET_POSTGRES_DBNAME
xyhw6mcr

xyhw6mcr1#

你需要将数据库标志添加到Postgres temp_file_limit中,使其更高,比如10mb,我认为它是二进制的,所以是10485760。如果您在复制副本上运行此命令,请添加temp_file_limit,但也要将max_standby_archive_delaymax_standby_streaming_delay添加到900s。

相关问题