首先，一个“全局”的监听器，对不同的组件做不同的事情，是个坏主意，它把太多的逻辑放在一个单一的组件中，把代码耦合起来，成为维护的噩梦。
话虽如此，你可以用一个 MouseListener 添加到每个组件，例如。。。

import java.awt.Color;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.GridLayout;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import java.awt.event.MouseListener;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.UIManager;
import javax.swing.UnsupportedLookAndFeelException;

public class Test {

    public static void main(String[] args) {
        new Test();
    }

    public Test() {
        EventQueue.invokeLater(new Runnable() {
            @Override
            public void run() {
                try {
                    UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
                } catch (ClassNotFoundException | InstantiationException | IllegalAccessException | UnsupportedLookAndFeelException ex) {
                    ex.printStackTrace();
                }

                JPanel left = new TestPane();
                JPanel right = new TestPane();

                left.setBackground(Color.RED);
                right.setBackground(Color.BLUE);

                left.setName("left");
                right.setName("right");

                MouseListener listener = new MouseAdapter() {
                    @Override
                    public void mouseClicked(MouseEvent e) {
                        System.out.println(((JPanel)e.getSource()).getName());
                    }
                };

                left.addMouseListener(listener);
                right.addMouseListener(listener);

                JFrame frame = new JFrame("Testing");
                frame.setLayout(new GridLayout(0, 2));
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                frame.add(left);
                frame.add(right);
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
            }
        });
    }

    public class TestPane extends JPanel {

        public TestPane() {
        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(200, 200);
        }

    }

}

那你就可以简单的使用了 MouseEvent#getSource 以确定触发事件的组件。为了简单起见，我提供了 name 对于每个面板，我会使用其他方法来确定组件，然后再决定要做什么。
更好的解决办法是提供一个具体的 MouseListener 它根据需要对每个面板做了特定的工作，这就变得更易于管理、隔离责任、分离代码并更易于维护和管理

编辑2
这应该是你的答案，我在调试时注意到，当我点击框架上的空白区域时，我得到一个名为 "null.contentPane" 所以我把它放在条件下，它就起作用了！，我希望我能帮上忙。

public class PanelsListener extends JFrame {

public static void main(String[] args) {
    PanelsListener pl = new PanelsListener();
    pl.setSize(700, 700);
    pl.setLayout(new FlowLayout());
    JPanel fp = new JPanel();

    fp.setName("First");
    fp.setSize(200, 200);
    fp.setBorder(new LineBorder(Color.red));
    JPanel sp = new JPanel();
    sp.setName("Second");
    sp.setSize(200, 200);
    sp.setBorder(new LineBorder(Color.blue));
    pl.add(fp);
    pl.add(sp);

    pl.addMouseListener(new MouseListener() {
        @Override
        public void mouseClicked(MouseEvent e) {
            JPanel pnl = (JPanel) javax.swing.SwingUtilities
                    .getDeepestComponentAt((PanelsListener) e.getSource(),
                            e.getX(), e.getY());
            if (pnl != null && (!pnl.getName().equals("null.contentPane"))) {
                String name = pnl.getName();
                if (name != null) {
                    if (name.equals("First")) {
                        pl.setSize(500, 500);
                    } else if (name.equals("Second")) {
                        pl.setSize(800, 800);
                    }
                    pl.repaint();
                }
            }

        }

        @Override
        public void mouseEntered(MouseEvent e) {
            // TODO Auto-generated method stub

        }

        @Override
        public void mouseExited(MouseEvent e) {
            // TODO Auto-generated method stub

        }

        @Override
        public void mousePressed(MouseEvent e) {
            // TODO Auto-generated method stub

        }

        @Override
        public void mouseReleased(MouseEvent e) {
            // TODO Auto-generated method stub

        }
    });

    pl.setVisible(true);
}

}

编辑1
我从你的评论中了解到，请尝试以下操作：

public class PanelsListener extends JFrame{

public static int  panel_identifier=0;

public static void main(String[] args) {
    PanelsListener pl = new PanelsListener();
    pl.setSize(700, 700);
    pl.setLayout(new FlowLayout());
    JPanel fp=new JPanel();
    fp.setSize(200,200);
    fp.setBorder(new LineBorder(Color.red));
    JPanel sp=new JPanel();
    sp.setSize(200,200);
    sp.setBorder(new LineBorder(Color.blue));

    pl.add(fp);
    pl.add(sp);

    //panel_identifier is used to specify which panel has been clicked.

    fp.addMouseListener(new MouseAdapter() {
        @Override
        public void mouseClicked(MouseEvent me) {
           pl.panel_identifier=1;
           pl.mouseClicked(me);
        }
    });
    sp.addMouseListener(new MouseAdapter() {
        @Override
        public void mouseClicked(MouseEvent me) {
            pl.panel_identifier=2;
             pl.mouseClicked(me);
        }
    });

    pl.setVisible(true);
}
protected void mouseClicked(MouseEvent me) {
    if(panel_identifier==1)
    {
        this.setSize(500, 500);
        panel_identifier=0;
    }
    else if(panel_identifier==2)
    {
        this.setSize(800, 800);
        panel_identifier=0;
    }

}
}

正常溶液
公共类panelslistener扩展了jframe{

public static void main(String[] args) {
    PanelsListener pl = new PanelsListener();
    pl.setSize(700, 700);
    pl.setLayout(new FlowLayout());
    pl.add(new FirstPanel(pl));
    pl.add(new SecondPanel(pl));
    pl.setVisible(true);
}
}

class FirstPanel extends JPanel {

public FirstPanel(Frame frm) {
    setBorder(new LineBorder(Color.red));
    setSize(200, 200);
    addMouseListener(new MouseAdapter() {
        @Override
        public void mouseClicked(MouseEvent e) {
            frm.setSize(500, 500);
            frm.repaint();
        }
    });
    addMouseMotionListener(new MouseAdapter() {
        @Override
        public void mouseDragged(MouseEvent e) {
            // do some action when mouse dragged over this panel
        }
    });
 }
 }

 class SecondPanel extends JPanel {

 public SecondPanel(Frame frm) {
    setBorder(new LineBorder(Color.blue));
    setSize(200, 200);
     addMouseListener(new MouseAdapter() {
     @Override
     public void mouseClicked(MouseEvent e) {
     frm.setSize(800,800);
     frm.repaint();
     }
     });

    addMouseMotionListener(new MouseAdapter() {
        @Override
        public void mouseDragged(MouseEvent e) {
            // do some action when mouse dragged over this panel
        }
    });
 }
 }

我不确定这是否是最好的方法，但我只是测试了它，它确实有效，至少就我的代码测试而言。当按下鼠标时，找出鼠标在哪个框上，并相应地执行操作。

public static void main(String[] args) {
    JFrame main = new JFrame("test");
    JPanel first = new JPanel();
    JPanel second = new JPanel();
    first.setSize(400, 500);
    first.setBackground(Color.BLUE);
    second.setBackground(Color.RED);
    second.setSize(400,500);
    main.setSize(800, 500);
    main.add(first);
    first.setLocation(0,0);
    second.setLocation(400, 500);
    main.add(second);
    main.pack();
    main.setVisible(true);
    main.setSize(800, 500);
    main.addMouseListener(new MouseListener() {

        @Override
        public void mousePressed(MouseEvent e) {
            if(first.getBounds().contains(new Point(e.getX(), e.getY())))
            {
                System.out.println("first box");
            }else if(second.getBounds().contains(new Point(e.getX(), e.getY())))
            {
                System.out.println("Second box");
            }
        }
    });
}