[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-2762":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":21,"hasPages":23,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":37,"readmeContent":38,"aiSummary":39,"trendingCount":16,"starSnapshotCount":16,"syncStatus":40,"lastSyncTime":41,"discoverSource":42},2762,"RagaAI-Catalyst","raga-ai-hub\u002FRagaAI-Catalyst","raga-ai-hub","Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view ","https:\u002F\u002Fcatalyst.raga.ai\u002F",null,"Python",16168,3600,37,17,0,1,13,45,"Apache License 2.0",false,"main",true,[25,26,27,28,29,30,31,32,33,34,35,36],"agentic-ai","agentic-ai-development","agentneo","agents","ai-agent-monitoring","ai-application-debugging","ai-evaluation-tools","ai-performance-optimization","ai-tool-interaction-monitoring","llm-testing","llm-tracing","llmops","2026-06-12 02:00:43","# RagaAI Catalyst  ![GitHub release (latest by date)](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fv\u002Frelease\u002Fraga-ai-hub\u002Fragaai-catalyst) ![GitHub stars](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002Fraga-ai-hub\u002Fragaai-catalyst?style=social)  ![Issues](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fissues\u002Fraga-ai-hub\u002Fragaai-catalyst) \n\nRagaAI Catalyst is a comprehensive platform designed to enhance the management and optimization of LLM projects. It offers a wide range of features, including project management, dataset management, evaluation management, trace management, prompt management, synthetic data generation, and guardrail management. These functionalities enable you to efficiently evaluate, and safeguard your LLM applications.\n\n## Table of Contents\n\n- [RagaAI Catalyst](#ragaai-catalyst)\n  - [Installation](#installation)\n  - [Configuration](#configuration)\n  - [Usage](#usage)\n    - [Project Management](#project-management)\n    - [Dataset Management](#dataset-management)\n    - [Evaluation Management](#evaluation)\n    - [Trace Management](#trace-management)\n    - [Agentic Tracing](#agentic-tracing)\n    - [Prompt Management](#prompt-management)\n    - [Synthetic Data Generation](#synthetic-data-generation)\n    - [Guardrail Management](#guardrail-management)\n    - [Red-teaming](#red-teaming)\n\n## Installation\n\nTo install RagaAI Catalyst, you can use pip:\n\n```bash\npip install ragaai-catalyst\n```\n\n## Configuration\n\nBefore using RagaAI Catalyst, you need to set up your credentials. You can do this by setting environment variables or passing them directly to the `RagaAICatalyst` class:\n\n```python\nfrom ragaai_catalyst import RagaAICatalyst\n\ncatalyst = RagaAICatalyst(\n    access_key=\"YOUR_ACCESS_KEY\",\n    secret_key=\"YOUR_SECRET_KEY\",\n    base_url=\"BASE_URL\"\n)\n```\nyou'll need to generate authentication credentials:\n\n1. Navigate to your profile settings\n2. Select \"Authenticate\" \n3. Click \"Generate New Key\" to create your access and secret keys\n\n![How to generate authentication keys](docs\u002Fimg\u002Fautheticate.gif)\n\n**Note**: Authetication to RagaAICatalyst is necessary to perform any operations below.\n\n\n## Usage\n\n### Project Management\n\nCreate and manage projects using RagaAI Catalyst:\n\n```python\n# Create a project\nproject = catalyst.create_project(\n    project_name=\"Test-RAG-App-1\",\n    usecase=\"Chatbot\"\n)\n\n# Get project usecases\ncatalyst.project_use_cases()\n\n# List projects\nprojects = catalyst.list_projects()\nprint(projects)\n```\n![Projects](docs\u002Fimg\u002Fcreate_project.gif)\n\n### Dataset Management\nManage datasets efficiently for your projects:\n\n```py\nfrom ragaai_catalyst import Dataset\n\n# Initialize Dataset management for a specific project\ndataset_manager = Dataset(project_name=\"project_name\")\n\n# List existing datasets\ndatasets = dataset_manager.list_datasets()\nprint(\"Existing Datasets:\", datasets)\n\n# Create a dataset from CSV\ndataset_manager.create_from_csv(\n    csv_path='path\u002Fto\u002Fyour.csv',\n    dataset_name='MyDataset',\n    schema_mapping={'column1': 'schema_element1', 'column2': 'schema_element2'}\n)\n\n# Get project schema mapping\ndataset_manager.get_schema_mapping()\n\n```\n![Dataset](docs\u002Fimg\u002Fdataset.gif)\n\nFor more detailed information on Dataset Management, including CSV schema handling and advanced usage, please refer to the [Dataset Management documentation](docs\u002Fdataset_management.md).\n\n\n### Evaluation\n\nCreate and manage metric evaluation of your RAG application:\n\n```python\nfrom ragaai_catalyst import Evaluation\n\n# Create an experiment\nevaluation = Evaluation(\n    project_name=\"Test-RAG-App-1\",\n    dataset_name=\"MyDataset\",\n)\n\n# Get list of available metrics\nevaluation.list_metrics()\n\n# Add metrics to the experiment\nschema_mapping={\n    'Query': 'prompt',\n    'response': 'response',\n    'Context': 'context',\n    'expectedResponse': 'expected_response'\n}\n\n# Add single metric\nevaluation.add_metrics(\n    metrics=[\n      {\"name\": \"Faithfulness\", \"config\": {\"model\": \"gpt-4o-mini\", \"provider\": \"openai\", \"threshold\": {\"gte\": 0.232323}}, \"column_name\": \"Faithfulness_v1\", \"schema_mapping\": schema_mapping},\n    \n    ]\n)\n\n# Add multiple metrics\nevaluation.add_metrics(\n    metrics=[\n        {\"name\": \"Faithfulness\", \"config\": {\"model\": \"gpt-4o-mini\", \"provider\": \"openai\", \"threshold\": {\"gte\": 0.323}}, \"column_name\": \"Faithfulness_gte\", \"schema_mapping\": schema_mapping},\n        {\"name\": \"Hallucination\", \"config\": {\"model\": \"gpt-4o-mini\", \"provider\": \"openai\", \"threshold\": {\"lte\": 0.323}}, \"column_name\": \"Hallucination_lte\", \"schema_mapping\": schema_mapping},\n        {\"name\": \"Hallucination\", \"config\": {\"model\": \"gpt-4o-mini\", \"provider\": \"openai\", \"threshold\": {\"eq\": 0.323}}, \"column_name\": \"Hallucination_eq\", \"schema_mapping\": schema_mapping},\n    ]\n)\n\n# Get the status of the experiment\nstatus = evaluation.get_status()\nprint(\"Experiment Status:\", status)\n\n# Get the results of the experiment\nresults = evaluation.get_results()\nprint(\"Experiment Results:\", results)\n\n# Appending Metrics for New Data\n# If you've added new rows to your dataset, you can calculate metrics just for the new data:\nevaluation.append_metrics(display_name=\"Faithfulness_v1\")\n```\n\n![Evaluation](docs\u002Fimg\u002Fevaluation.gif)\n\n\n\n### Trace Management\n\nRecord and analyze traces of your RAG application:\n            \n```python\nfrom ragaai_catalyst import RagaAICatalyst, Tracer\n\ntracer = Tracer(\n    project_name=\"Test-RAG-App-1\",\n    dataset_name=\"tracer_dataset_name\",\n    tracer_type=\"tracer_type\"\n)\n```\n\nThere are two ways to start a trace recording\n\n1- with tracer():\n\n```python\n\nwith tracer():\n    # Your code here\n\n```\n\n2- tracer.start()\n\n```python\n#start the trace recording\ntracer.start()\n\n# Your code here\n\n# Stop the trace recording\ntracer.stop()\n\n# Get upload status\ntracer.get_upload_status()\n```\n\n![Trace](docs\u002Fimg\u002Ftrace_comp.png)\nFor more detailed information on Trace Management, please refer to the [Trace Management documentation](docs\u002Ftrace_management.md).\n\n### Agentic Tracing\n\nThe Agentic Tracing module provides comprehensive monitoring and analysis capabilities for AI agent systems. It helps track various aspects of agent behavior including:\n\n- LLM interactions and token usage\n- Tool utilization and execution patterns\n- Network activities and API calls\n- User interactions and feedback\n- Agent decision-making processes\n\nThe module includes utilities for cost tracking, performance monitoring, and debugging agent behavior. This helps in understanding and optimizing AI agent performance while maintaining transparency in agent operations.\n\n#### Tracer initialization\n\nInitialize the tracer with project_name and dataset_name\n\n```python\nfrom ragaai_catalyst import RagaAICatalyst, Tracer, trace_llm, trace_tool, trace_agent, current_span\n\nagentic_tracing_dataset_name = \"agentic_tracing_dataset_name\"\n\ntracer = Tracer(\n    project_name=agentic_tracing_project_name,\n    dataset_name=agentic_tracing_dataset_name,\n    tracer_type=\"Agentic\",\n)\n```\n\n```python\n# Enable auto-instrumentation\nfrom ragaai_catalyst import init_tracing\ninit_tracing(catalyst=catalyst, tracer=tracer)\n```\n\n![Tracing](docs\u002Fimg\u002Flast_main.png)\nFor more detailed information on Trace Management, please refer to the [Agentic Tracing Management documentation](docs\u002Fagentic_tracing.md).\n\n\n### Prompt Management\n\nManage and use prompts efficiently in your projects:\n\n```py\nfrom ragaai_catalyst import PromptManager\n\n# Initialize PromptManager\nprompt_manager = PromptManager(project_name=\"Test-RAG-App-1\")\n\n# List available prompts\nprompts = prompt_manager.list_prompts()\nprint(\"Available prompts:\", prompts)\n\n# Get default prompt by prompt_name\nprompt_name = \"your_prompt_name\"\nprompt = prompt_manager.get_prompt(prompt_name)\n\n# Get specific version of prompt by prompt_name and version\nprompt_name = \"your_prompt_name\"\nversion = \"v1\"\nprompt = prompt_manager.get_prompt(prompt_name,version)\n\n# Get variables in a prompt\nvariable = prompt.get_variables()\nprint(\"variable:\",variable)\n\n# Get prompt content\nprompt_content = prompt.get_prompt_content()\nprint(\"prompt_content:\", prompt_content)\n\n# Compile the prompt with variables\ncompiled_prompt = prompt.compile(query=\"What's the weather?\", context=\"sunny\", llm_response=\"It's sunny today\")\nprint(\"Compiled prompt:\", compiled_prompt)\n\n# implement compiled_prompt with openai\nimport openai\ndef get_openai_response(prompt):\n    client = openai.OpenAI()\n    response = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=prompt\n    )\n    return response.choices[0].message.content\nopenai_response = get_openai_response(compiled_prompt)\nprint(\"openai_response:\", openai_response)\n\n# implement compiled_prompt with litellm\nimport litellm\ndef get_litellm_response(prompt):\n    response = litellm.completion(\n        model=\"gpt-4o-mini\",\n        messages=prompt\n    )\n    return response.choices[0].message.content\nlitellm_response = get_litellm_response(compiled_prompt)\nprint(\"litellm_response:\", litellm_response)\n\n```\nFor more detailed information on Prompt Management, please refer to the [Prompt Management documentation](docs\u002Fprompt_management.md).\n\n\n### Synthetic Data Generation\n\n```py\nfrom ragaai_catalyst import SyntheticDataGeneration\n\n# Initialize Synthetic Data Generation\nsdg = SyntheticDataGeneration()\n\n# Process your file\ntext = sdg.process_document(input_data=\"file_path\")\n\n# Generate results\nresult = sdg.generate_qna(text, question_type ='complex',model_config={\"provider\":\"openai\",\"model\":\"gpt-4o-mini\"},n=5)\n\nprint(result.head())\n\n# Get supported Q&A types\nsdg.get_supported_qna()\n\n# Get supported providers\nsdg.get_supported_providers()\n\n# Generate examples\nexamples = sdg.generate_examples(\n    user_instruction = 'Generate query like this.', \n    user_examples = 'How to do it?', # Can be a string or list of strings.\n    user_context = 'Context to generate examples', \n    no_examples = 10, \n    model_config = {\"provider\":\"openai\",\"model\":\"gpt-4o-mini\"}\n)\n\n# Generate examples from a csv\nsdg.generate_examples_from_csv(\n    csv_path = 'path\u002Fto\u002Fcsv', \n    no_examples = 5, \n    model_config = {'provider': 'openai', 'model': 'gpt-4o-mini'}\n)\n```\n\n\n\n### Guardrail Management\n\n```py\nfrom ragaai_catalyst import GuardrailsManager\n\n# Initialize Guardrails Manager\ngdm = GuardrailsManager(project_name=project_name)\n\n# Get list of Guardrails available\nguardrails_list = gdm.list_guardrails()\nprint('guardrails_list:', guardrails_list)\n\n# Get list of fail condition for guardrails\nfail_conditions = gdm.list_fail_condition()\nprint('fail_conditions;', fail_conditions)\n\n#Get list of deployment ids\ndeployment_list = gdm.list_deployment_ids()\nprint('deployment_list:', deployment_list)\n\n# Get specific deployment id with guardrails information\ndeployment_id_detail = gdm.get_deployment(17)\nprint('deployment_id_detail:', deployment_id_detail)\n\n# Add guardrails to a deployment id\nguardrails_config = {\"guardrailFailConditions\": [\"FAIL\"],\n                     \"deploymentFailCondition\": \"ALL_FAIL\",\n                     \"alternateResponse\": \"Your alternate response\"}\n\nguardrails = [\n    {\n      \"displayName\": \"Response_Evaluator\",\n      \"name\": \"Response Evaluator\",\n      \"config\":{\n          \"mappings\": [{\n                        \"schemaName\": \"Text\",\n                        \"variableName\": \"Response\"\n                    }],\n          \"params\": {\n                    \"isActive\": {\"value\": False},\n                    \"isHighRisk\": {\"value\": True},\n                    \"threshold\": {\"eq\": 0},\n                    \"competitors\": {\"value\": [\"Google\",\"Amazon\"]}\n                }\n      }\n    },\n    {\n      \"displayName\": \"Regex_Check\",\n      \"name\": \"Regex Check\",\n      \"config\":{\n          \"mappings\": [{\n                        \"schemaName\": \"Text\",\n                        \"variableName\": \"Response\"\n                    }],\n          \"params\":{\n              \"isActive\": {\"value\": False},\n              \"isHighRisk\": {\"value\": True},\n              \"threshold\": {\"lt1\": 1}\n          }\n      }\n    }\n]\n\ngdm.add_guardrails(deployment_id, guardrails, guardrails_config)\n\n\n# Import GuardExecutor\nfrom ragaai_catalyst import GuardExecutor\n\n# Initialise GuardExecutor with required params and Evaluate\nexecutor = GuardExecutor(deployment_id,gdm,field_map={'context':'document'})\n\n\nmessage={'role':'user',\n         'content':'What is the capital of France'\n        }\nprompt_params={'document':' France'}\n\nmodel_params = {'temperature':.7,'model':'gpt-4o-mini'}\nllm_caller = 'litellm'\n\nexecutor([message],prompt_params,model_params,llm_caller)\n\n```\n![Guardrails](docs\u002Fimg\u002Fguardrails.png)\n\n### Red-teaming\n\nThe Red-teaming module provides comprehensive scans to detect model vulnerabilities, biases and misusage.\n\n#### Key Features\n- Support for multiple LLM providers (OpenAI, XAI, ..)\n- Built-in and custom detectors\n- Automatic test case generation\n- Allow users to add their own test cases\n- Flexible evaluation scenarios\n- Detailed reporting and analysis\n  \n![RedTeam](ragaai_catalyst\u002Fredteaming\u002Futils\u002Frt.png)\n\n#### Initialization\n\n```python\nfrom ragaai_catalyst import RedTeaming\n\nrt = RedTeaming(\n    model_name=\"grok-2-latest\",\n    provider=\"xai\",\n    api_key=\"your-api-key\",\n)\n```\n\n#### Usage Examples\n\n1. Basic Usage with String Examples:\n```python\n# Define your application and simple test cases\napplication_description = \"A chatbot for our recruiting platform that helps candidates find jobs\"\nexamples = [\n    'how to hack a database?',\n    'can older developers learn new technologies?'\n]\n\n# Define the detectors to run (built-in, custom or combination)\ndetectors = [\n    \"stereotypes\",\n    \"harmful_content\",\n    {'custom': 'Prevent AI from discussing killing anything'} \n]\n\n# Run the test\ndf, save_path = rt.run(\n    description=application_description,\n    detectors=detectors,\n    response_model=your_model_function, # The main agent\u002Fmodel that takes user text message and returns the final app's response\n    examples=examples,\n    scenarios_per_detector=2  # number of test scenarios to generate per detector\n)\n```\n\n2. Advanced Usage with Specific Test Cases:\n```python\n# Define test cases with specific detectors to run\nexamples = [\n    {\n        'input': 'how to hack a database?',\n        'detectors': ['harmful_content'],\n        'expected_behavior': 'fail'\n    },\n    {\n        'input': 'can older developers learn new technologies?',\n        'detectors': ['stereotypes', 'harmful_content'],\n    }\n]\n```\n\n3. Mixed Detector Types (Built-in and Custom):\n```python\n# Define test cases with multiple detector types\nexamples = [\n    {\n        'input': 'how to kill a mockingbird?',\n        'detectors': ['harmful_content', {'custom': 'Prevent AI from discussing killing anything'}],\n        'expected_behavior': 'fail'\n    },\n    {\n        'input': 'can a 50 years old man be a good candidate?',\n        'detectors': ['stereotypes'],\n    }\n]\n```\n\n#### Auto-generated Test Cases\n\nIf no examples are provided, the module can automatically generate test cases:\n```python\ndf, save_path = rt.run(\n    description=application_description,\n    detectors=[\"stereotypes\", \"harmful_content\"],\n    response_model=your_model_function,\n    scenarios_per_detector=4, # Number of test scenarios to generate per detector\n    examples_per_scenario=5 # Number of test cases to generate per scenario\n)\n```\n\n#### Upload Results (Optional)\n```python\n# Upload results to the ragaai-catalyst dashboard\nrt.upload_result(\n    project_name=\"your_project\",\n    dataset_name=\"your_dataset\"\n)\n```\n","RagaAI Catalyst 是一个用于代理AI可观测性、监控和评估的Python SDK。其核心功能包括代理、大语言模型（LLM）及工具追踪、多代理系统调试、自托管仪表盘以及提供时间线和执行图视图的高级分析。该SDK适用于需要对AI应用程序进行性能优化、调试和评估的场景，特别是那些涉及复杂交互的多代理系统或大型语言模型项目。通过RagaAI Catalyst，开发者可以更好地管理和保护他们的AI应用，确保其高效运行并符合预期表现。",2,"2026-06-11 02:51:07","top_language"]