The code is as follows. I have a highly imbalanced dataset for chest x rays with heart enlargement. The images are separated into a training folder split into positive for cardiomegaly and negative for cardiomegaly subfolders (467 pos images and ~20,000 neg). (Then I have a testing folder with two subfolders (300 pos, 300 neg). Each time I test I keep getting a 50% accuracy with the eval method below. When I look at the predictions it is always that they are all one class (normally negative), however if I give the positive values a very high weight (1000+ compared to the negative values 1) the model will flip and say that they are all instead positive. This leads me to believe it is overfitting but all my attempts to resolve this have come up with issues.

import pandas as pd
import os
import matplotlib.pyplot as plt
import numpy as np
import skimage as sk
import skimage.io as skio
import skimage.transform as sktr
import skimage.filters as skfl
import skimage.feature as skft
import skimage.color as skcol
import skimage.exposure as skexp
import skimage.morphology as skmr
import skimage.util as skut
import skimage.measure as skme
import sklearn.model_selection as le_ms
import sklearn.decomposition as le_de
import sklearn.discriminant_analysis as le_di
import sklearn.preprocessing as le_pr
import sklearn.linear_model as le_lm
import sklearn.metrics as le_me
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
classNames = ["trainpos","trainneg"]
testclassNames = ["testpos", "test"]
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    './data/trainup/',
    labels='inferred',
    label_mode='categorical',
    class_names=classNames,
    color_mode='grayscale',
    batch_size=32,
    image_size=(256, 256),
    shuffle=True,
    seed=123,
    validation_split=0.2,
    subset="training",
    interpolation='gaussian',
    follow_links=False,
)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    './data/trainup/',
    labels='inferred',
    label_mode='categorical',
    class_names=classNames,
    color_mode='grayscale',
    batch_size=32,
    image_size=(256, 256),
    shuffle=True,
    seed=23,
    validation_split=0.2,
    subset="validation",
    interpolation='gaussian',
    follow_links=False,
)

test_ds = tf.keras.preprocessing.image_dataset_from_directory(
    './data/testup/',
    labels='inferred',
    label_mode='categorical',
    class_names=testclassNames,
    color_mode='grayscale',
    batch_size=32,
    image_size=(256, 256),
    shuffle=True,
    interpolation='gaussian',
    follow_links=False,
)

AUTOTUNE = tf.data.experimental.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)

model = tf.keras.Sequential([
  tf.keras.layers.experimental.preprocessing.Rescaling(1./255, input_shape=(256, 256, 1)),
  tf.keras.layers.Conv2D(16, 4, padding='same', activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Conv2D(32, 4, padding='same', activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(2)
])

opt = keras.optimizers.Adam(learning_rate=0.0001)
model.compile(optimizer=opt,
              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

class_weight = {0: 29, 1: 1}

history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=5,
   class_weight=class_weight
)
test_loss, test_accuracy = model.evaluate(test_ds)
print("Test Loss: ", test_loss)
print("Test Accuracy: ", test_accuracy)

19/19 [==============================] - 7s 376ms/step - loss: 3.4121 - accuracy: 0.5000 Test Loss: 3.4121198654174805 Test Accuracy: 0.5
I have tried updating the learning rate to values between 0.1 and 0.00001, adding epochs, removing epochs, changing to SGP for the optimizer, attempting to unpack the test_ds after subscripting it gave me the error that it is a batchdataset and can't be subscripted. This then shows me that the test_ds is giving me ~19 tensors of 32 images each except the last one which has about 25. I then wanted to predict each of these images individually and get the results because it looked like it was grouping all 32 (or 25 for the last one) together and then predicting based on that but that led me down rabbitholes that I haven't come out of with results. Tried many other things I can't fully remember normally tweaking the model itself or adding data augmentation (I am using tensorflow 2.3 as this is for a class with a repeating assignment so the data augmentation cannot be done with the current docs (mostly just vertical and horizontal changes in this version from what I can tell)