[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80096":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":13,"stars7d":16,"stars30d":17,"stars90d":15,"forks30d":15,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":21,"hasPages":21,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":28,"readmeContent":29,"aiSummary":30,"trendingCount":15,"starSnapshotCount":15,"syncStatus":16,"lastSyncTime":31,"discoverSource":32},80096,"MedMemoryBench","AQ-MedAI\u002FMedMemoryBench","AQ-MedAI","The source code and data link of paper \"MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare\".","https:\u002F\u002Fx.com\u002FAQ_MedAI",null,"Python",60,1,57,0,2,4,3,0.9,"Apache License 2.0",false,"main",[24,25,26,27],"agent-memory","benchmark","llm","medical-ai","2026-06-12 02:03:58","# MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\"figs\u002F4examples.png\" alt=\"MedMemoryBench overview — four key challenges in medical memory evaluation\" width=\"800\"\u002F>\n\u003C\u002Fdiv>\n\n\u003Cp align=\"center\">\n  \u003Cem>Solving the issues of agent memory evaluation in healthcare scenarios.\u003C\u002Fem>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n｜🤗 \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FCyan27\u002FMedMemoryBench\" target=\"_blank\">HuggingFace Dataset\u003C\u002Fa> ｜\n📄 \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.11814\">Arxiv Preprint\u003C\u002Fa> ｜\n🌐 \u003Ca href=\"README_ZH.md\">中文\u003C\u002Fa> ｜\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fversion-1.0.0-blue\" alt=\"version\"\u002F>\n  \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fpython-%3E%3D3.10-blue\" alt=\"python\"\u002F>\n  \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-Apache%202.0-green\" alt=\"license\"\u002F>\n\u003C\u002Fp>\n\n---\n\n**MedMemoryBench** is a benchmark framework for evaluating Agent memory methods, with a focus on memory capability assessment in medical dialogue scenarios. This framework provides unified evaluation interfaces, multiple baseline method implementations, and a flexible configuration management system, while also supporting the import and evaluation of other datasets.\n\n## Table of Contents\n\n- [News](#-news)\n- [Features](#-features)\n- [Project Structure](#-project-structure)\n- [Quick Start](#-quick-start)\n- [Configuration](#-configuration)\n- [Output](#-output)\n- [Citations](#-citations)\n\n---\n\n## 📰 News\n\n- **[2026.05]** MedMemoryBench v1.0 is officially released — dataset, evaluation framework, and 14 memory method baselines.\n- **[2026.05]** Dataset available on [HuggingFace](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FCyan27\u002FMedMemoryBench).\n\n## ✨ Features\n\n\u003Ctable>\n\u003Ctr>\n\u003Ctd width=\"50%\">\n\n**Comprehensive Medical Dataset**\n- 20 longitudinal patient personas with background, life events, and trap events\n- ~2,020 multi-session doctor–patient dialogue sessions\n- ~1,986 evaluation queries across 6 clinically motivated types\n- Bilingual support: Chinese (~598 MB) + English (~443 MB)\n\n\u003C\u002Ftd>\n\u003Ctd width=\"50%\">\n\n**Rich Baseline Coverage**\n- **3 classic baselines**: Long Context, Embedding RAG, BM25 RAG\n- **7 agentic memory systems**: Mem0, Letta, MemOS, A-MEM, MIRIX, MemRL, LightMem\n- **4 graph-based systems**: GraphRAG, HippoRAG-v2, ReMem, Zep\n\n\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\n\n**Unified Evaluation Framework**\n- Plug-and-play method integration via `BaseAgent`\n- Multi-metric evaluation: string match + LLM-as-a-Judge\n- Checkpoint & resume for long-running experiments\n- Dry-run mode for fast pipeline validation\n\n\u003C\u002Ftd>\n\u003Ctd>\n\n**Flexible Configuration**\n- YAML-driven method & dataset configs\n- Multi-provider LLM support (OpenAI \u002F BigModel \u002F Azure)\n- Local & remote embedding models\n- Cross-benchmark evaluation (MedMemoryBench + LoCoMo)\n\n\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftable>\n\n## 📁 Project Structure\n\n\u003Cdetails>\n\u003Csummary>Click to expand full directory tree\u003C\u002Fsummary>\n\n```\nMedMemoryBench\u002F\n├── main.py                       # Evaluation entry point\n├── requirements.txt              # Python dependencies\n├── LICENSE                       # Apache License 2.0\n├── LEGAL.md                      # Comment-language legal notice\n├── .env.example                  # Environment variable template\n│\n├── configs\u002F                      # Configuration files\n│   ├── method_config\u002F            # Per-method YAML configs (gpt-5.1 \u002F qwen3 variants)\n│   │   ├── long_context_gpt-5.1.yaml\n│   │   ├── embedding_rag_gpt-5.1.yaml\n│   │   ├── bm25_rag_gpt-5.1.yaml\n│   │   ├── graph_rag_gpt-5.1.yaml\n│   │   ├── mem0_gpt-5.1.yaml\n│   │   ├── memos_gpt-5.1.yaml\n│   │   ├── memrl_gpt-5.1.yaml\n│   │   ├── amem_gpt-5.1.yaml\n│   │   ├── hipporag_gpt-5.1.yaml\n│   │   ├── lightmem_gpt-5.1.yaml\n│   │   ├── letta_gpt-5.1.yaml\n│   │   ├── mirix_gpt-5.1.yaml\n│   │   ├── remem_gpt-5.1.yaml\n│   │   ├── zep_gpt-5.1-chat.yaml\n│   │   └── ...                   # + qwen3 variants\n│   └── dataset_config\u002F\n│       ├── medmemorybench.yaml\n│       └── locomo.yaml\n│\n├── methods\u002F                      # Memory method implementations\n│   ├── base.py                   # BaseAgent abstract class\n│   ├── long_context.py           # Long-context baseline\n│   ├── embedding_rag.py          # Dense embedding RAG\n│   ├── bm25_rag.py               # BM25 sparse RAG\n│   ├── graph_rag.py              # Graph-based RAG\n│   ├── self_rag.py               # Self-RAG\n│   ├── mem0_agent.py             # Mem0 adapter\n│   ├── memos_agent.py            # MemOS adapter\n│   ├── memrl_agent.py            # MemRL adapter\n│   ├── amem_agent.py             # A-MEM adapter\n│   ├── hipporag_agent.py         # HippoRAG adapter\n│   ├── lightmem_agent.py         # LightMem adapter\n│   ├── letta_agent.py            # Letta adapter\n│   ├── mirix_agent.py            # MIRIX adapter\n│   ├── remem_agent.py            # ReMem adapter\n│   ├── zep_agent.py              # Zep Cloud adapter\n│   └── \u003Cvendored repos>          # mem0\u002F, memOS\u002F, MemRL\u002F, amem\u002F, HippoRAG\u002F,\n│                                 # LightMem\u002F, letta\u002F, MIRIX\u002F, REMem\u002F, MEM1\u002F,\n│                                 # cognee\u002F, memorag\u002F  (third-party sources)\n│\n├── benchmarks\u002F                   # Dataset evaluation implementations\n│   ├── base.py                   # BaseDataset abstract class\n│   ├── medmemorybench\u002F           # MedMemoryBench dataset\n│   │   ├── dataset.py\n│   │   ├── evaluator.py\n│   │   └── checkpoint.py\n│   └── locomo\u002F                   # LoCoMo dataset\n│       ├── dataset.py\n│       └── evaluator.py\n│\n├── metrics\u002F                      # Evaluation metrics\n│   ├── base.py                   # BaseMetric abstract class\n│   ├── string_match.py           # String matching metrics\n│   ├── llm_judge.py              # LLM-as-a-Judge metrics\n│   └── locomo_metrics.py         # LoCoMo-specific metrics\n│\n├── src\u002F                          # Core orchestration modules\n│   ├── config.py                 # Configuration loader\n│   ├── agent.py                  # AgentManager\n│   ├── evaluator.py              # Evaluation dispatcher\n│   └── result.py                 # Result collection & reporting\n│\n├── utils\u002F                        # Utility modules\n│   ├── llm_client.py             # Unified LLM client\n│   ├── tokenizer.py              # Tokenizer helpers\n│   ├── templates.py              # Prompt templates\n│   ├── prompts_qa.py             # QA prompts\n│   ├── prompts_judge.py          # Judge prompts\n│   ├── prompts_memorize.py       # Memorization prompts\n│   ├── langchain_callback.py     # LangChain callback hooks\n│   └── logger.py                 # Logger\n│\n├── docker\u002F                       # Optional service compose files\n│   ├── mirix-init.sql\n│   └── mirix-services.yml\n│\n├── scripts\u002F                      # Helper scripts\n│   ├── run_eval.sh\n│   └── mirix-services.sh\n│\n├── data\u002F                         # Datasets (Git LFS)\n│   ├── MedMemoryBench\u002F           # Chinese, ~598 MB\n│   ├── MedMemoryBench_EN\u002F        # English, ~443 MB\n│   └── locomo\u002F                   # LoCoMo, ~18 MB\n│\n├── generation\u002F                   # Dataset generation pipeline (sub-project)\n├── outputs\u002F                      # Evaluation outputs (gitignored)\n├── exp_results\u002F                  # Curated experiment reports\n├── logs\u002F                         # Runtime logs (gitignored)\n└── results\u002F                      # Method-side caches (gitignored)\n```\n\n\u003C\u002Fdetails>\n\n## 🚀 Quick Start\n\n### 1. Clone the Repository\n\n> **Note:** This repository ships datasets via **Git LFS**. Please install it before cloning.\n\n```bash\n# Install Git LFS (skip if already installed)\nbrew install git-lfs                  # macOS\nsudo apt-get install git-lfs          # Ubuntu\u002FDebian\n# Windows: https:\u002F\u002Fgit-lfs.github.com\u002F\n\ngit lfs install\ngit clone https:\u002F\u002Fgithub.com\u002FAQ-MedAI\u002FMedMemoryBench.git\ncd MedMemoryBench\n```\n\n### 2. Environment Setup\n\n\u003Cdetails open>\n\u003Csummary>\u003Cb>Using uv (recommended)\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\ncurl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002Finstall.sh | sh\n\nuv venv\nsource .venv\u002Fbin\u002Factivate          # Linux\u002FmacOS\n# .venv\\Scripts\\activate           # Windows\n\nuv pip install -r requirements.txt\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cb>Using conda\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\nconda create -n medmemorybench python=3.10\nconda activate medmemorybench\npip install -r requirements.txt\n```\n\n\u003C\u002Fdetails>\n\n> **Method-specific dependencies:** Some memory methods vendor upstream packages under `methods\u002F` (e.g. `methods\u002Fmem0\u002F`, `methods\u002FmemOS\u002F`). If a method has its own `requirements.txt` or `README`, follow those instructions to enable it.\n\n> **Embedding models:** Method configs reference local embedding models or API. For the former, please download the embedded model before running.\n\n### 3. Configure Environment Variables\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` and fill in the API keys you intend to use:\n\n```env\n# BigModel (OpenAI-compatible, primary endpoint used in this project)\nBIGMODEL_API_KEY=your_bigmodel_api_key\nBIGMODEL_BASE_URL=https:\u002F\u002Fopen.bigmodel.cn\u002Fapi\u002Fpaas\u002Fv4\n\n# OpenAI (optional)\nOPENAI_API_KEY=your_openai_api_key\nOPENAI_BASE_URL=https:\u002F\u002Fapi.openai.com\u002Fv1\n\n# Azure OpenAI (optional)\nAZURE_OPENAI_API_KEY=your_azure_key\nAZURE_OPENAI_ENDPOINT=https:\u002F\u002Fyour-endpoint.openai.azure.com\u002F\n\n# Zep Cloud (optional, only needed for the Zep agent)\nZEP_API_KEY=your_zep_api_key\n\n# Default model selection\nDEFAULT_LLM_MODEL=gpt-4o-mini\nDEFAULT_EMBEDDING_MODEL=text-embedding-3-small\nEMBEDDING_PROVIDER=openai\n\n# Optional: isolate Letta local runtime data (defaults to ~\u002F.letta)\nLETTA_DIR=.tmp\u002Fletta_runtime\n```\n\n> **Tips:**\n> - For BigModel, set `BIGMODEL_API_KEY` \u002F `BIGMODEL_BASE_URL` first; the framework maps them to OpenAI-compatible settings internally.\n> - `LETTA_DIR` is recommended to avoid stale SQLite metadata from previous Letta runs.\n\n### 4. Run Evaluation\n\n**Via shell script:**\n\n```bash\n.\u002Fscripts\u002Frun_eval.sh bm25_rag_gpt-5.1 medmemorybench\n```\n\n**Via Python:**\n\n```bash\n# Standard run\npython main.py -m bm25_rag_gpt-5.1 -d medmemorybench\n\n# Dry run (no real LLM\u002FAPI calls)\npython main.py -m embedding_rag_gpt-5.1 -d medmemorybench --dry-run\n\n# Resume from checkpoint\npython main.py -m embedding_rag_gpt-5.1 -d medmemorybench --resume\n```\n\n\u003C!-- > 💡 **Extending with a new method?** See [`methods\u002FREADME.md`](methods\u002FREADME.md) for the step-by-step guide. -->\n\n## 🔧 Configuration\n\n### Method Configuration\n\nEach method is driven by a YAML file under `configs\u002Fmethod_config\u002F`:\n\n```yaml\n# configs\u002Fmethod_config\u002Fembedding_rag_gpt-5.1.yaml\n\nmethod_name: \"embedding_rag\"\nmethod_type: \"rag\"                  # baseline \u002F rag \u002F agentic_memory\ndescription: \"Embedding RAG Agent - Dense vector retrieval based RAG method\"\n\nmodel:\n  provider: \"openai\"\n  name: \"gpt-5.1\"\n  temperature: 0.3\n  max_completion_tokens: 100000\n\nagent_params:\n  top_k: 5                          # Number of documents to retrieve\n  chunk_size: 512                   # Text chunk size\n  chunk_overlap: 50                 # Chunk overlap\n\nembedding:\n  provider: \"local\"                 # openai \u002F local \u002F huggingface\n  model: \"\u002Fpath\u002Fto\u002Flocal\u002Fmodel\"\n```\n\n### Dataset Configuration\n\nDataset configs live under `configs\u002Fdataset_config\u002F`:\n\n```yaml\n# configs\u002Fdataset_config\u002Fmedmemorybench.yaml\n\ndataset_name: \"medmemorybench\"\ndescription: \"Medical dialogue memory evaluation dataset\"\nlanguage: \"zh\"\n\ndata:\n  root_dir: \"data\u002FMedMemoryBench\"\n  sessions_pattern: \"persona_{id}\u002Feval\u002Fgenerated_dialogues.json\"\n  queries_pattern: \"persona_{id}\u002Feval\u002Fgenerated_queries.json\"\n\nevaluation:\n  mode: \"independent\"               # independent \u002F merged\n  evaluation_interval: 10           # Evaluate every N sessions\n\nquery_types:\n  - name: \"entity_exact_match\"\n    metric: \"string_contain\"\n  - name: \"temporal_localization\"\n    metric: \"llm_judge\"\n  # ... more types\n```\n\n## 📄 Output\n\nEvaluation results are saved under `outputs\u002F\u003Cmethod>_\u003Cmodel>\u002F`:\n\n```\noutputs\u002F\n└── bm25_rag_gpt-5.1\u002F\n    ├── eval_medmemorybench_20260330_181703.json    # Detailed results (JSON)\n    ├── report_medmemorybench_20260330_181703.txt   # Human-readable report\n    └── memory_builds_20260330_181703.json          # Memory build logs\n```\n\n## 📝 Citations\n\nIf you find MedMemoryBench useful in your research, please consider citing our work:\n\n```bibtex\n@article{wang2026medmemorybench,\n  title={MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare},\n  author={Yihao Wang and Haoran Xu and Renjie Gu and Yixuan Ye and Xinyi Chen and Xinyu Mu and Yuan Gao and Chunxiao Guo and Peng Wei and Jinjie Gu and Huan Li and Ke Chen and Lidan Shou},\n  journal={arXiv preprint arXiv:2605.11814},\n  year={2026}\n}\n```\n\n---\n\n## 📜 License\n\n- **Code** — [Apache License 2.0](LICENSE)\n- **Dataset** (`data\u002FMedMemoryBench\u002F`, `data\u002FMedMemoryBench_EN\u002F`) — [CC BY 4.0](https:\u002F\u002Fcreativecommons.org\u002Flicenses\u002Fby\u002F4.0\u002F)\n- **Vendored third-party sources** under `methods\u002F` retain their original upstream licenses.\n- See [LEGAL.md](LEGAL.md) for the source-comment language clause.\n","MedMemoryBench是一个专注于医疗对话场景中代理记忆能力评估的基准框架。它提供统一的评估接口、多种基线方法实现以及灵活的配置管理系统，支持导入和评估其他数据集。该框架涵盖了20个纵向患者角色的数据集，包含约2,020个多会话医患对话记录，并支持中文和英文双语环境。其丰富的基线覆盖包括3种经典基线、7种代理记忆系统及4种基于图的方法。此外，MedMemoryBench还具备多指标评估功能，如字符串匹配与大模型作为裁判，同时支持实验的检查点保存与恢复，非常适合需要在个性化医疗服务中测试或改进记忆代理性能的研究者和开发者使用。","2026-06-11 03:59:14","CREATED_QUERY"]