[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-10765":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":15,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":19,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":34,"readmeContent":35,"aiSummary":36,"trendingCount":16,"starSnapshotCount":16,"syncStatus":37,"lastSyncTime":38,"discoverSource":39},10765,"nlp_xiaojiang","yongzhuo\u002Fnlp_xiaojiang","yongzhuo","自然语言处理（nlp），小姜机器人（闲聊检索式chatbot），BERT句向量-相似度（Sentence Similarity），XLNET句向量-相似度（text xlnet embedding），文本分类（Text classification）， 实体提取（ner，bert+bilstm+crf），数据增强（text augment, data enhance），同义句同义词生成，句子主干提取（mainpart），中文汉语短文本相似度，文本特征工程，keras-http-service调用","https:\u002F\u002Fblog.csdn.net\u002Frensihui",null,"Python",1537,391,40,1,0,55.88,"MIT License",false,"master",true,[23,24,25,26,27,28,29,30,31,32,33],"bert","chatbot","chinese","data-augmentation","distance","enhance","feature","nlp","text-augment","text-classification","xlnet","2026-06-12 04:00:52","# nlp_xiaojiang\n\n\n# AugmentText\n    - 回译（效果比较好）\n    - EDA（同义词替换、插入、交换和删除）（效果还行）\n    - HMM-marko（质量较差）\n    - syntax（依存句法、句法、语法书）（简单句还可）\n    - seq2seq（深度学习同义句生成，效果不理想，seq2seq代码大都是 [https:\u002F\u002Fgithub.com\u002Fqhduan\u002Fjust_another_seq2seq] 的，效果不理想）\n    - 预训练(UNILM生成、开源模型回译)\n    \n# ChatBot\n    - 检索式ChatBot\n        - 像ES那样直接检索(如使用fuzzywuzzy)，只能字面匹配\n        - 构造句向量，检索问答库，能够检索有同义词的句子\n    - 生成式ChatBot（todo）\n        - seq2seq\n        - GAN\n\n# ClassificationText\n    - bert+bi-lstm(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n    - bert + text-cnn(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n    - bert + r-cnn(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n    - bert + avt-cnn(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n\n# Ner\n    - bert命名实体提取(bert12层embedding + bilstm + crf)\n        - args.py(配置一些参数)\n        - keras_bert_embedding.py(bert embedding)\n        - keras_bert_layer.py(layer层, 主要有CRF和NonMaskingLayer)\n        - keras_bert_ner_bi_lstm.py(主函数, 定义模型、数据预处理和训练预测等)\n        - layer_crf_bojone.py(CRF层, 未使用)\n\n# FeatureProject\n    - bert句向量、文本相似度\n        - bert\u002Fextract_keras_bert_feature.py:提取bert句向量特征\n        - bert\u002Ftet_bert_keras_sim.py:测试xlnet句向量cosin相似度\n    - xlnet句向量、文本相似度\n        - xlnet\u002Fextract_keras_xlnet_feature.py:提取bert句向量特征\n        - xlnet\u002Ftet_xlnet_keras_sim.py:测试bert句向量cosin相似度\n    - normalization_util指的是数据归一化\n        - 0-1归一化处理\n        - 均值归一化\n        - sig归一化处理\n    - sim feature（ML）\n        - distance_text_or_vec:各种计算文本、向量距离等\n        - distance_vec_TS_SS：TS_SS计算词向量距离\n        - cut_td_idf：将小黄鸡语料和gossip结合\n        - sentence_sim_feature：计算两个文本的相似度或者距离，例如qq（问题和问题），或者qa（问题和答案）\n\n# run(可以在win10下,pycharm下运行)\n  - 1.创建tf-idf文件等（运行2需要先跑1）:      \n                                       ```\n                                       python cut_td_idf.py\n                                       ```\n  - 2.计算两个句子间的各种相似度，先计算一个预定义的，然后可输入自定义的（先跑1）:  \n                                       ```\n                                       python sentence_sim_feature.py\n                                       ```\n  - 3.chatbot_1跑起来(fuzzy检索-没)（独立）：    \n                                       ```\n                                       python chatbot_fuzzy.py\n                                       ```\n  - 4.chatbot_2跑起来(句向量检索-词)（独立）：    \n                                       ```\n                                       python chatbot_sentence_vec_by_word.py\n                                       ```\n  - 5.chatbot_3跑起来(句向量检索-字)（独立）：    \n                                       ```\n                                       python chatbot_sentence_vec_by_char.py\n                                       ```\n  - 6.数据增强（eda)：                     python enhance_eda.py\n  - 7.数据增强（marko）:                   python enhance_marko.py\n  - 8.数据增强（translate_account）:       python translate_tencent_secret.py\n  - 9.数据增强（translate_tools）:         python translate_translate.py\n  - 10.数据增强（translate_web）:          python translate_google.py\n  - 11.数据增强（augment_seq2seq）:        先跑 python extract_char_webank.py生成数据，\n                                          再跑 python train_char_anti.py\n                                          然后跑 python predict_char_anti.py\n  - 12.特征计算(bert)（提取特征、计算相似度）: \n                      ```\n                      run extract_keras_bert_feature.py\n                      run tet_bert_keras_sim.py\n                      ```\n                      \n# Data\n    - chinese_L-12_H-768_A-12（谷歌预训练好的模型）\n       github项目中只是上传部分数据，需要的前往链接: https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1I3vydhmFEQ9nuPG2fDou8Q 提取码: rket\n       解压后就可以啦\n    - chinese_xlnet_mid_L-24_H-768_A-12(哈工大训练的中文xlnet, mid, 24层, wiki语料+通用语料)\n        - 下载地址[https:\u002F\u002Fgithub.com\u002Fymcui\u002FChinese-PreTrained-XLNet](https:\u002F\u002Fgithub.com\u002Fymcui\u002FChinese-PreTrained-XLNet)\n    - chinese_vector\n        github项目中只是上传部分数据，需要的前往链接: https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1I3vydhmFEQ9nuPG2fDou8Q 提取码: rket\n        - 截取的部分word2vec训练词向量（自己需要下载全效果才会好）\n        - w2v_model_wiki_char.vec、w2v_model_wiki_word.vec都只有部分，词向量w2v_model_wiki_word.vec可以用这个下载地址的替换[https:\u002F\u002Fpan.baidu.com\u002Fs\u002F14JP1gD7hcmsWdSpTvA3vKA](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F14JP1gD7hcmsWdSpTvA3vKA)\n\n    - corpus\n        github项目中只是上传部分数据，需要的前往链接: https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1I3vydhmFEQ9nuPG2fDou8Q 提取码: rket\n        - ner(train、dev、test----人民日报语料)\n        - webank(train、dev、test)\n        - 小黄鸡和gossip问答预料（数据没清洗）,chicken_and_gossip.txt\n        - 微众银行和支付宝文本相似度竞赛数据， sim_webank.csv\n    - sentence_vec_encode_char\n        - 1.txt（字向量生成的前100000句向量）\n    - sentence_vec_encode_word\n        - 1.txt（词向量生成的前100000句向量）\n    - tf_idf（chicken_and_gossip.txt生成的tf-idf）\n    \n# requestments.txt\n    - python_Levenshtei\n        - 调用Levenshtein，我的python是3.6，\n        - 打开其源文件: https:\u002F\u002Fwww.lfd.uci.edu\u002F~gohlke\u002Fpythonlibs\u002F\n        - 查找python_Levenshtein-0.12.0-cp36-cp36m-win_amd64.whl下载即可\n    - pyemd\n    - pyhanlp\n        - 下好依赖JPype1-0.6.3-cp36-cp36m-win_amd64.whl\n  \n# 参考\u002F感谢\n* eda_chinese：[https:\u002F\u002Fgithub.com\u002Fzhanlaoban\u002Feda_nlp_for_Chinese](https:\u002F\u002Fgithub.com\u002Fzhanlaoban\u002Feda_nlp_for_Chinese)\n* 主谓宾提取器：[https:\u002F\u002Fgithub.com\u002Fhankcs\u002FMainPartExtractor](https:\u002F\u002Fgithub.com\u002Fhankcs\u002FMainPartExtractor)\n* HMM生成句子：[https:\u002F\u002Fgithub.com\u002FtakeToDreamLand\u002FSentenceGenerate_byMarkov](https:\u002F\u002Fgithub.com\u002FtakeToDreamLand\u002FSentenceGenerate_byMarkov)\n* 同义词等：[https:\u002F\u002Fgithub.com\u002Ffighting41love\u002FfunNLP\u002Ftree\u002Fmaster\u002Fdata\u002F](https:\u002F\u002Fgithub.com\u002Ffighting41love\u002FfunNLP\u002Ftree\u002Fmaster\u002Fdata\u002F)\n* 小牛翻译：[http:\u002F\u002Fwww.niutrans.com\u002Findex.html](http:\u002F\u002Fwww.niutrans.com\u002Findex.html)\n    \n# 其他资料\n* bert(keras):[https:\u002F\u002Fgithub.com\u002FCyberZHG\u002Fkeras-bert](https:\u002F\u002Fgithub.com\u002FCyberZHG\u002Fkeras-bert)\n* NLP数据增强汇总:[https:\u002F\u002Fgithub.com\u002Fquincyliang\u002Fnlp-data-augmentation](https:\u002F\u002Fgithub.com\u002Fquincyliang\u002Fnlp-data-augmentation)\n* 知乎NLP数据增强话题:[https:\u002F\u002Fwww.zhihu.com\u002Fquestion\u002F305256736\u002Fanswer\u002F550873100](https:\u002F\u002Fwww.zhihu.com\u002Fquestion\u002F305256736\u002Fanswer\u002F550873100)\n* chatbot_seq2seq_seqGan（比较好用）：[https:\u002F\u002Fgithub.com\u002Fqhduan\u002Fjust_another_seq2seq](https:\u002F\u002Fgithub.com\u002Fqhduan\u002Fjust_another_seq2seq)\n* 自己动手做聊天机器人教程: [https:\u002F\u002Fgithub.com\u002Fwarmheartli\u002FChatBotCourse](https:\u002F\u002Fgithub.com\u002Fwarmheartli\u002FChatBotCourse)\n\n","nlp_xiaojiang 是一个专注于自然语言处理的开源项目，主要实现了一个基于检索式的闲聊机器人小姜。该项目利用 BERT 和 XLNET 模型生成句向量来计算文本相似度，并支持文本分类、实体识别等功能。通过使用 Keras 构建深度学习模型，如 BERT+BiLSTM 或 BERT+CRF，实现了较高的准确率。此外，它还提供了多种数据增强方法，包括回译和 EDA 等技术以提高模型训练效果。适合于需要构建聊天机器人、进行文本分析或者希望探索 NLP 相关算法的研究者及开发者使用。",2,"2026-06-11 03:30:03","top_topic"]