如何解决使用转换器程序包加载BertModel时遇到“ UnicodeDecodeError”
我下载了经过预训练的文件格式google，并使用官方脚本convert_bert_original_tf_checkpoint_to_pytorch.py将其转换为名称为'pytorch_model.bin'的火炬版：
current_dir = os.path.dirname(__file__) # (1)
model_folder = os.path.join(current_dir,'chinese_L-12_H-768_A-12')
checkpoint = os.path.join(model_folder,'bert_model.ckpt')
configer = os.path.join(model_folder,'bert_config.json')
save_to = os.path.join(current_dir,'torch bert model','pytorch_model.bin')
convert_tf_checkpoint_to_pytorch(checkpoint,configer,save_to)
然后，我将“ bert_config.json”重命名为“ config.json”，并将“ pytorch_model.bin”，“ config.json”和“ vocab.txt”放入相同的文件夹“ torch_bert_model”中，以便加载该手电筒'from_pertained（）'方法的模型：
current_dir = os.path.dirname(__file__)
model_folder = os.path.join(current_dir,'torch_bert_model')
# configer_file = os.path.join(model_folder,'config.json')
vocab_file = os.path.join(model_folder,'vocab.txt')
model_file = os.path.join(model_folder,'pytorch_model.bin')
tokenizer = BertTokenizer.from_pretrained(vocab_file)
# bert_config = BertConfig.from_json_file(configer_file)
model = TFBertModel.from_pretrained(model_file) # Automatically loads the config
运行此代码时，发生了“ UnicodeDecodeError”：
Traceback (most recent call last):
File "c:/Users/hap/Desktop/Projects/cgt/BERT.py",line 47,in <module>
model = TFBertModel.from_pretrained(model_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site- packages\transformers\modeling_tf_utils.py",line 529,in from_pretrained
config,model_kwargs = cls.config_class.from_pretrained(
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py",line 311,in from_pretrained
config_dict,kwargs = cls.get_config_dict(pretrained_model_name_or_path,**kwargs)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py",line 354,in get_config_dict
config_dict = cls._dict_from_json_file(resolved_config_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py",line 436,in _dict_from_json_file
text = reader.read()
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\codecs.py",line 322,in decode
(result,consumed) = self._buffer_decode(data,self.errors,final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte
我在哪里错了？任何建议，不胜感激。谢谢！
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。