langchain-ai/langsmith-skills
3 skills9.6k installs426 starsGitHub
Install
npx skills add https://github.com/langchain-ai/langsmith-skillsSkills in this repo
1Langsmith Tracelangsmith-trace is a LangSmith skill covering two topics: adding tracing to applications and querying traces for debugging and analysis in Python and JavaScript. Setup requires LANGSMITH_API_KEY and optionally LANGSMITH_PROJECT and LANGSMITH_WORKSPACE_ID; CLI commands accept --api-key when env vars are unset. For LangChain and LangGraph apps, tracing is automatic once LANGSMITH_TRACING and LANGSMITH_API_KEY are exported, with LANGCHAIN_CALLBACKS_BACKGROUND=false recommended for serverless. Non-LangChain apps use OpenTelemetry when available or the traceable decorator plus wrap_openai for Python client instrumentation with nested span examples for RAG pipelines. The langsmith CLI installs via curl from langchain-ai/langsmith-cli and supports trace list and export operations against named projects. The skill stresses checking LANGSMITH_PROJECT in the environment before querying so agents target the correct trace store. Developers invoke it when instrumenting LLM pipelines or pulling production traces for latency, failure, and prompt debugging.3.2kinstalls2Langsmith EvaluatorLangSmith Evaluator is a LangChain skill for building agent evaluation pipelines across three components: creating evaluators, defining run functions, and running evaluations locally or via uploaded evaluators. Offline evaluators attach to datasets with a (run, example) signature; online evaluators attach to projects with a (run) signature for production checks. The golden rule requires running the agent on sample inputs and inspecting actual output before writing evaluator logic. LLM-as-Judge evaluators use structured output; custom code evaluators handle trajectory validation and format checks. Each evaluator returns one metric only as score and comment. Python local evaluate() passes RunTree objects while uploaded evaluators receive dicts, so handlers must support both. Run functions capture outputs and optionally trajectories via LangGraph stream_mode debug with subgraphs. The langsmith CLI uploads code evaluators that auto-run on dataset experiments. Uploaded evaluators run in a sandbox with limited packages, so local evaluate() is preferred for development. Setup requires LANGSMITH_API_KEY and langsmith plus langchain-openai dependencies.3.2kinstalls3Langsmith Datasetlangsmith-dataset is an agent skill for building and managing LangSmith evaluation datasets used to test agent behavior. It documents required LANGSMITH_API_KEY setup, optional LANGSMITH_PROJECT and LANGSMITH_WORKSPACE_ID variables, and CLI installation from the langsmith-cli install script. Dataset types include final_response for full conversations, single_step for one node, trajectory for ordered tool call sequences, and rag for question, chunks, answer, and citations structures. Workflows cover exporting traces with langsmith trace export, processing JSONL roots into inputs and outputs examples, uploading JSON arrays via langsmith dataset upload, and creating examples directly through the langsmith Python or JavaScript Client. CLI commands list, get, create, delete, export, and upload datasets plus example and experiment management with safety prompts that require user confirmation unless running non-interactively. Troubleshooting addresses invalid JSON shapes, empty uploads, missing --full trace exports, and example count mismatches. Developers reach for it when creating evaluation datasets, uploading local JSON to LangSmith, or managing examples for agent testing and validat.3.1kinstalls