How to improve neural network accuracy for binary classification

Other Questions ai employee
1

I’m working on a binary classification problem and trying to optimize my model performance. My dataset has around 16k samples with 7 features each.

Current setup:

  • Total parameters: approximately 48K
  • Input data shape: (samples, 7)
  • Binary output (0 or 1)

Sample input data:

data_x = np.array([[ 2019,    15, 45123, ...,     0,  7234,     0],
                   [ 2019,    15, 45123, ...,     0,  7899,     0],
                   [ 2019,    31, 45123, ...,     0,  7234,     0],
                   ...,
                   [ 8500,    25, 52341, ...,     0,  9876,     0]], dtype=int32)

Target labels:

labels_y = np.array([0., 0., 0., ..., 1., 0., 0.])

My current model architecture:

neural_net = Sequential()
neural_net.add(BatchNormalization(input_shape=(7,)))
neural_net.add(Dense(64, activation="relu"))
neural_net.add(Dense(64, activation="relu"))
neural_net.add(Dense(128, activation="relu"))
neural_net.add(Dense(32, activation="relu"))
neural_net.add(Dense(1, activation='sigmoid'))

Training configuration:

neural_net.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
training_history = neural_net.fit(train_x, train_y, batch_size=1024, 
                                 validation_data=(test_x, test_y), epochs=30000)

I’ve tried different approaches:

  • Adding batch normalization improved accuracy from 50% to 73%
  • Tested both ‘adam’ and ‘rmsprop’ optimizers with similar results
  • Experimented with different batch sizes (1024, 2048)
  • Extended training to 30000 epochs which gave me 78.51% accuracy
  • Doubled layer sizes but still getting similar performance

Currently getting around 6k correct predictions out of 16k samples. What techniques or modifications should I try to push the accuracy higher?

2

Feature engineering’s probably your bottleneck. I had the same problem with mixed categorical/numerical features - the model just couldn’t find patterns. With only 7 features, you’ve got room to create interaction features or polynomial combos without worrying about overfitting. Also check if you’re treating categorical features (like year 2019) as numerical. Converting them to embeddings or one-hot encoding usually works way better than raw integers. And honestly? Try XGBoost or LightGBM first - neural networks often suck for tabular data with few features.

3

your model looks solid. check for class imbalance tho - if most samples are class 0, that 78% accuracy might just mean it’s predicting everything as 0. look at precision/recall instead of just accuracy. also, add dropout layers between your dense layers (0.3-0.5 rate) to help with overfitting.

4

30,000 epochs is way too much - you’re definitely overfitting. I ran into the same thing last year on a classification project where more training actually made results worse. Set up early stopping with patience around 50-100 epochs and watch validation loss, not just accuracy. Your input data is all over the place (small numbers mixed with 45123+) which messes with training. StandardScaler or MinMaxScaler fixed this for me. With only 7 features, your network’s probably too complex. I’d try 2 hidden layers with 32-64 neurons each - simpler usually works better with small feature sets.

5

I’ve hit this exact problem with tabular data. Your learning rate’s too high for that dataset size.

Drop it to 0.001 or 0.0001 instead of Adam’s default. I was stuck at the same accuracy on a fraud model until I cut the learning rate and added scheduling.

from tensorflow.keras.callbacks import ReduceLROnPlateau

reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=10, min_lr=1e-7)

optimizer = Adam(learning_rate=0.001)
neural_net.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])

That batch size of 1024 is way too big for 16k samples. Use 64 or 128. Smaller batches are noisier but usually find better solutions.

If you’ve got class imbalance, swap binary crossentropy for focal loss. Took me from 76% to 84% on a similar project by targeting the hard examples my model kept screwing up.