bert 评估结果不同

yws3nbqq  于 4个月前  发布在  其他
关注(0)|答案(4)|浏览(58)

当我使用相同的评估集在没有"do_train"和"do_train, do_eval"的情况下运行"do eval"时，出现了两个不同的结果，前者比后者低得多。是否有任何设置错误？

bert

4条答案

按热度按时间
j2qf4p5b

j2qf4p5b1#

训练和评估：

python run_classifier.py 
--data_dir=$DATA_DIR 
--task_name=xxx 
--do_train=true 
--do_eval=true 
--do_predict=false 
--vocab_file=$BERT_BASE_DIR/vocab.txt 
--bert_config_file=$BERT_BASE_DIR/bert_config.json 
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt 
--max_seq_length=32 
--train_batch_size=32 
--learning_rate=2e-5 
--num_train_epochs=3.0 
--output_dir=../output

得到结果：
...
INFO:tensorflow:Saving checkpoints for 306482 into .../output/model.ckpt.
INFO:tensorflow:Loss for final step: 0.68741745.
INFO:tensorflow:training_loop marked as finished
INFO:tensorflow:Writing example 0 of 181618
INFO:tensorflow:*** Example ***
...
INFO:tensorflow:Finished evaluation at 2019-03-05-17:12:46
INFO:tensorflow:Saving dict for global step 306482: eval_accuracy = 0.7659318, eval_loss = 1.156508, global_step = 306482, loss = 1.1564742
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 306482: ../output/model.ckpt-306482
INFO:tensorflow:evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.7659318
INFO:tensorflow: eval_loss = 1.156508
INFO:tensorflow: global_step = 306482
INFO:tensorflow: loss = 1.1564742
之后，我运行

python run_classifier.py 
--data_dir=$DATA_DIR 
--task_name=xxx 
--do_eval=true 
--do_predict=false 
--vocab_file=$BERT_BASE_DIR/vocab.txt 
--bert_config_file=$BERT_BASE_DIR/bert_config.json 
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt 
--max_seq_length=32 
--train_batch_size=32 
--learning_rate=2e-5 
--num_train_epochs=3.0 
--output_dir=../output

得到不同的结果：
INFO:tensorflow:Finished evaluation at 2019-03-06-02:57:04
INFO:tensorflow:Saving dict for global step 306482: eval_accuracy = 0.0013820216, eval_loss = 8.9281435, global_step = 306482, loss = 8.928147
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 306482: ../output/model.ckpt-306482
INFO:tensorflow:evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.0013820216
INFO:tensorflow: eval_loss = 8.9281435
INFO:tensorflow: global_step = 306482
INFO:tensorflow: loss = 8.928147

赞(0）分享  回复(0）举报  4个月前
yi0zb3m4

yi0zb3m42#

我也有同样的问题...
训练和评估：

python run_classifier.py
--data_dir=$DATA_DIR
--task_name=xxx
--do_train=true
--do_eval=true
--do_predict=false
--vocab_file=$BERT_BASE_DIR/vocab.txt
--bert_config_file=$BERT_BASE_DIR/bert_config.json
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt
--max_seq_length=32
--train_batch_size=32
--learning_rate=2e-5
--num_train_epochs=3.0
--output_dir=../output

得到结果：
...
INFO:tensorflow:Saving checkpoints for 306482 into .../output/model.ckpt.
INFO:tensorflow:Loss for final step: 0.68741745.
INFO:tensorflow:training_loop marked as finished
INFO:tensorflow:Writing example 0 of 181618
INFO:tensorflow:*** Example ***
...
INFO:tensorflow:Finished evaluation at 2019-03-05-17:12:46
INFO:tensorflow:Saving dict for global step 306482: eval_accuracy = 0.7659318, eval_loss = 1.156508, global_step = 306482, loss = 1.1564742
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 306482: ../output/model.ckpt-306482
INFO:tensorflow:evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.7659318
INFO:tensorflow: eval_loss = 1.156508
INFO:tensorflow: global_step = 306482
INFO:tensorflow: loss = 1.1564742
之后，我运行了以下命令：

python run_classifier.py
--data_dir=$DATA_DIR
--task_name=xxx
--do_eval=true
--do_predict=false
--vocab_file=$BERT_BASE_DIR/vocab.txt
--bert_config_file=$BERT_BASE_DIR/bert_config.json
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt
--max_seq_length=32
--train_batch_size=32
--learning_rate=2e-5
--num_train_epochs=3.0
--output_dir=../output

得到了不同的结果：
INFO:tensorflow:Finished evaluation at 2019-03-06-02:57:04
INFO:tensorflow:Saving dict for global step 306482: eval_accuracy = 0.0013820216, eval_loss = 8.9281435, global_step = 306482, loss = 8.928147
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 306482: ../output/model.ckpt-306482
INFO:tensorflow:evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.0013820216
INFO:tensorflow: eval_loss = 8.9281435
INFO:tensorflow: global_step = 306482
INFO:tensorflow: loss = 8.928147

赞(0）分享  回复(0）举报  4个月前
niwlg2el

niwlg2el3#

在训练过程中，模型会对训练集进行 Shuffle ，因此如果将训练和评估过程分开，训练标签列表和评估标签列表之间可能会出现不匹配。你可以保存训练标签列表，并在评估模型时重用它。

要恢复训练标签列表，你可以使用train.tf_record文件。但是，重新训练可能需要花费一定的时间。

赞(0）分享  回复(0）举报  4个月前
iecba09b

iecba09b4#

训练和评估：

python run_classifier.py
--data_dir=$DATA_DIR
--task_name=xxx
--do_train=true
--do_eval=true
--do_predict=false
--vocab_file=$BERT_BASE_DIR/vocab.txt
--bert_config_file=$BERT_BASE_DIR/bert_config.json
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt
--max_seq_length=32
--train_batch_size=32
--learning_rate=2e-5
--num_train_epochs=3.0
--output_dir=../output

得到结果：
...
INFO:tensorflow:Saving checkpoints for 306482 into .../output/model.ckpt.
INFO:tensorflow:Loss for final step: 0.68741745.
INFO:tensorflow:training_loop marked as finished
INFO:tensorflow:Writing example 0 of 181618
INFO:tensorflow:*** Example ***
...
INFO:tensorflow:Finished evaluation at 2019-03-05-17:12:46
INFO:tensorflow:Saving dict for global step 306482: eval_accuracy = 0.7659318, eval_loss = 1.156508, global_step = 306482, loss = 1.1564742
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 306482: ../output/model.ckpt-306482
INFO:tensorflow:evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.7659318
INFO:tensorflow: eval_loss = 1.156508
INFO:tensorflow: global_step = 306482
INFO:tensorflow: loss = 1.1564742
之后，我运行了以下命令：

python run_classifier.py
--data_dir=$DATA_DIR
--task_name=xxx
--do_eval=true
--do_predict=false
--vocab_file=$BERT_BASE_DIR/vocab.txt
--bert_config_file=$BERT_BASE_DIR/bert_config.json
--init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt
--max_seq_length=32
--train_batch_size=32
--learning_rate=2e-5
--num_train_epochs=3.0
--output_dir=../output

得到了不同的结果：
INFO:tensorflow:Finished evaluation at 2019-03-06-02:57:04
INFO:tensorflow:Saving dict for global step 306482: eval_accuracy = 0.0013820216, eval_loss = 8.9281435, global_step = 306482, loss = 8.928147
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 306482: ../output/model.ckpt-306482
INFO:tensorflow:evaluation_loop marked as finished
INFO:tensorflow:***** Eval results *****
INFO:tensorflow: eval_accuracy = 0.0013820216
INFO:tensorflow: eval_loss = 8.9281435
INFO:tensorflow: global_step = 306482
INFO:tensorflow: loss = 8.928147
在训练过程中，模型会打乱训练集，因此如果将训练和评估过程分开进行，训练标签列表和评估标签列表之间可能会出现不匹配。你可以保存训练标签列表，并在评估模型时重用它。谢谢！

赞(0）分享  回复(0）举报  4个月前
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com