我在运行Meltano时遇到了这个问题,使用标签和目标“postgresql”作为源数据库,而没有通过点击“postgresql”来更改temp_file_limit参数。我该怎么弥补?psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (13411002kB)
Meltanomeltano.yml设置文件如下所示:
default_environment: dev
project_id: fake-id
plugins:
extractors:
- name: tap-postgres
variant: transferwise
pip_url: pipelinewise-tap-postgres
- name: tap-google-analytics
variant: meltanolabs
pip_url: pip install git+https://github.com/MeltanoLabs/tap-google-analytics.git
loaders:
- name: target-postgres
variant: transferwise
pip_url: pipelinewise-target-postgres
config:
schema_mapping:
schema_1:
target_schema: replica_schema_1
default_target_schema: public
orchestrators:
- name: airflow
pip_url: psycopg2 apache-airflow==2.1.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt
files:
- name: airflow
pip_url: git+https://gitlab.com/meltano/files-airflow.git
schedules:
- name: schedule-name
extractor: tap-postgres
loader: target-postgres
transform: skip
interval: '*/30 * * * *'
environments:
- name: dev
config:
plugins:
extractors:
- name: tap-postgres
config:
dbname: $TAP_POSTGRES_DBNAME
user: $TAP_POSTGRES_USER
select:
- schema_name-table_name_1
- schema_name-table_name_2
loaders:
- name: target-postgres
config:
user: $TARGET_POSTGRES_USER
dbname: $TARGET_POSTGRES_DBNAME
1条答案
按热度按时间xyhw6mcr1#
你需要将数据库标志添加到Postgres
temp_file_limit
中,使其更高,比如10mb,我认为它是二进制的,所以是10485760。如果您在复制副本上运行此命令,请添加temp_file_limit
,但也要将max_standby_archive_delay
和max_standby_streaming_delay
添加到900s。