我正在使用Keras（VGG 16、VGG 19、Xception......）的预训练模型对4000多张图像（带YOLO坐标）的数据集进行对象检测模型训练，以下是负责训练和验证数据的数据预处理以及模型编译和训练的部分。
对于VGG 16和VGG 19-我调整图像和YOLO坐标到推荐的默认图像大小224 x224，而对于Xception和InceptionV 3，我调整到299 x299。
我冻结了Keras应用程序的所有层，只添加了4个顶级Dense层，这些层在我的数据集上进行训练，以利用预训练模型的潜力。当我使用VGG 16或VGG 19时，它运行良好，我最终的训练和验证准确率超过92%，这很棒，训练/验证分割似乎是平衡的。但是当我使用Xception或InceptionV 3应用程序时，它总是以10%左右的准确率提前停止，这我不明白。

IMAGE_SIZE = (299, 299)

train_datagen = ImageDataGenerator(rescale=1./255, validation_split=0.2)  # val 20%

val_datagen = ImageDataGenerator(rescale=1./255, validation_split=0.2)

train_data = train_datagen.flow_from_dataframe(
    dataframe=df_all, 
    directory=save_dir,                                               
    x_col="image_name", 
    y_col=['yolo_x', 'yolo_y', 'yolo_width', 'yolo_length'], 
    class_mode="raw", 
    target_size=IMAGE_SIZE,
    batch_size=32,
    shuffle=True,
    Subset='training'
)

val_data = val_datagen.flow_from_dataframe(
    dataframe=df_all, 
    directory=save_dir,                                               
    x_col="image_name", 
    y_col=['yolo_x', 'yolo_y', 'yolo_width', 'yolo_length'], 
    class_mode="raw", 
    target_size=IMAGE_SIZE,
    batch_size=32,
    shuffle=False,
    Subset='validation'
)

from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint

learning_rate_reduction = ReduceLROnPlateau(monitor='loss', 
                                            patience=5, 
                                            verbose=2, 
                                            factor=0.5,                                            
                                            min_lr=0.000001)

early_stops = EarlyStopping(monitor='loss', 
                            min_delta=0, 
                            patience=10, 
                            verbose=2, 
                            mode='auto')

checkpointer = ModelCheckpoint(filepath = 'cis3115.{epoch:02d}-{accuracy:.6f}.hdf5',
                               verbose=2,
                               save_best_only=True, 
                               save_weights_only = True)

# Select a pre-trained model Xception
#pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])
pretrained_model = tf.keras.applications.Xception(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])

# Set the following to False so that the pre-trained weights are not changed 
pretrained_model.trainable = False 

model = Sequential()
model.add(pretrained_model)

# Flatten 2D images into 1D data for final layers like traditional neural network
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))

# The final output layer
# Use Sigmoid when predicting YOLO bounding box since that output is between 0 and 1
model.add(Dense(4, activation='sigmoid'))

print ("Pretrained model used:")
pretrained_model.summary()

print ("Final model created:")
model.summary()

# Compile neural network model
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])

# Train the model with the images in the folders
history = model.fit(
        train_data,
        validation_data=val_data,
        batch_size=16,                  # Number of image batches to process per epoch 
        epochs=100,                      # Number of epochs
        callbacks=[learning_rate_reduction, early_stops],
        )

Xception是一个复杂得多的预训练模型，因此理论上应该更精确，因此我假设我在设置CNN时做错了什么。
Xception / Inception模型失败的原因是什么？我应该改变什么？