Langfuse JS/TS SDKs
    Preparing search index...

    Class ExperimentManager

    Manages the execution and evaluation of experiments on datasets.

    The ExperimentManager provides a comprehensive framework for running experiments that test models or tasks against datasets, with support for automatic evaluation, scoring.

    const langfuse = new LangfuseClient();

    const result = await langfuse.experiment.run({
      name: "Capital Cities Test",
      description: "Testing model knowledge of world capitals",
      data: [
        { input: "France", expectedOutput: "Paris" },
        { input: "Germany", expectedOutput: "Berlin" }
      ],
      task: async ({ input }) => {
        const response = await openai.chat.completions.create({
          model: "gpt-4",
          messages: [{ role: "user", content: `What is the capital of ${input}?` }]
        });
        return response.choices[0].message.content;
      },
      evaluators: [
        async ({ input, output, expectedOutput }) => ({
          name: "exact_match",
          value: output === expectedOutput ? 1 : 0
        })
      ]
    });

    console.log(await result.format());

    const dataset = await langfuse.dataset.get("my-dataset");

    const result = await dataset.runExperiment({
      name: "Model Comparison",
      task: myTask,
      evaluators: [myEvaluator],
      runEvaluators: [averageScoreEvaluator]
    });
    Index

    Constructors

    constructor

    Accessors

    logger

    Methods

    run

    Constructors

    • Internal

      Creates a new ExperimentManager instance.

      Parameters

      • params: { langfuseClient: LangfuseClient }

        Configuration object

        • langfuseClient: LangfuseClient

          The Langfuse client instance for API communication

      Returns ExperimentManager

    Accessors

    • get logger(): Logger
      Internal

      Gets the global logger instance for experiment-related logging.

      Returns Logger

      The global logger instance

    Methods

    • Executes an experiment by running a task on each data item and evaluating the results.

      This method orchestrates the complete experiment lifecycle:

      1. Executes the task function on each data item with proper tracing
      2. Runs item-level evaluators on each task output
      3. Executes run-level evaluators on the complete result set
      4. Groups all items into one Langfuse experiment (stable per dataset and run name)
      5. Stores all scores and traces in Langfuse

      Type Parameters

      • Input = any
      • ExpectedOutput = any
      • Metadata extends Record<string, any> = Record<string, any>

      Parameters

      • config: ExperimentParams<Input, ExpectedOutput, Metadata>

        The experiment configuration

        • data: ExperimentItem<Input, ExpectedOutput, Metadata>[]

          Array of data items to process.

          Can be either custom ExperimentItem[] or DatasetItem[] from Langfuse. Each item should contain input data and optionally expected output.

        • OptionaldatasetVersion?: string

          Optional ISO 8601 timestamp (RFC 3339, Section 5.6) in UTC (e.g., "2026-01-21T14:35:42Z"). If provided, returns state of dataset at this timestamp. If not provided, returns the latest version. Recorded on each dataset item's observation as the experiment item version.

        • Optionaldescription?: string

          Optional description explaining the experiment's purpose.

          Provide context about what you're testing, methodology, or goals. This helps with experiment tracking and result interpretation.

        • Optionalevaluators?: Evaluator<Input, ExpectedOutput, Metadata>[]

          Optional array of evaluator functions to assess each item's output.

          Each evaluator receives input, output, and expected output (if available) and returns evaluation results. Multiple evaluators enable comprehensive assessment.

        • OptionalexperimentId?: string

          Optional ID of the Langfuse experiment to record this run into.

          Langfuse groups experiment items by this ID, so separate run calls that pass the same ID (for example from parallel workers, or when resuming an interrupted run) land in one experiment. Use it for local data, which otherwise gets a new random ID on every call.

          When set, it is used as-is for every item of the run, including items from Langfuse datasets, and as the datasetRunId of run-level scores. Re-runs on a dataset with the same runName then join this ID rather than the derived one. When omitted, runs on Langfuse datasets derive a stable ID from the project, dataset and runName, and runs on local data get a random ID. Must not be empty or whitespace-only.

          Pass the same runName on every worker as well; otherwise each one records its own timestamped run name.

          Generate the ID once with createTraceId from @langfuse/tracing and share it with every worker, or derive it on each worker from a seed they all know. Seeded IDs match the Python SDK's create_trace_id(seed=...), so Python and JS workers can share one experiment:

          import { createTraceId } from "@langfuse/tracing";

          // Random: create once, then pass to every worker
          const experimentId = await createTraceId();

          // Deterministic: the same seed yields the same ID on every worker
          const experimentId = await createTraceId("nightly-eval-2026-10-20");

          await langfuse.experiment.run({
            name: "Nightly eval",
            runName: "nightly-eval-2026-10-20",
            experimentId,
            data: shardForThisWorker,
            task,
          });

          Re-running an item records a new attempt: the item view shows the latest attempt, but run-level aggregates such as cost and score averages include every attempt. Run-level evaluators only see the items of their own run call.

        • OptionalmaxConcurrency?: number

          Maximum number of concurrent task executions (default: Infinity).

          Controls parallelism to manage resource usage and API rate limits. Set lower values for expensive operations or rate-limited services.

        • Optionalmetadata?: Record<string, any>

          Optional metadata to attach to the experiment run.

          Store additional context like model versions, hyperparameters, or any other relevant information for analysis and comparison.

        • name: string

          Human-readable name for the experiment.

          This name will appear in Langfuse UI and experiment results. Choose a descriptive name that identifies the experiment's purpose.

        • OptionalrunEvaluators?: RunEvaluator<Input, ExpectedOutput, Metadata>[]

          Optional array of run-level evaluators to assess the entire experiment.

          These evaluators receive all item results and can perform aggregate analysis like calculating averages, detecting patterns, or statistical analysis.

        • OptionalrunName?: string

          Optional exact name for the experiment run.

          If provided, this will be used as the exact dataset run name if the data contains Langfuse dataset items. If not provided, this will default to the experiment name appended with an ISO timestamp.

        • task: ExperimentTask<Input, ExpectedOutput, Metadata>

          The task function to execute on each data item.

          This function receives input data and produces output that will be evaluated. It should encapsulate the model or system being tested.

      Returns Promise<ExperimentResult<Input, ExpectedOutput, Metadata>>

      Promise that resolves to experiment results including:

      • experimentId: Identifier of the experiment in Langfuse
      • experimentUrl: Link to the experiment results in the Langfuse UI
      • runName: The experiment run name (either provided or generated)
      • itemResults: Results for each processed data item
      • runEvaluations: Results from run-level evaluators
      • datasetRunId: ID of the dataset run (if using Langfuse datasets)
      • format: Function to format results for display

      When task execution fails and cannot be handled gracefully

      When required evaluators fail critically

      const result = await langfuse.experiment.run({
        name: "Translation Quality Test",
        data: [
          { input: "Hello world", expectedOutput: "Hola mundo" },
          { input: "Good morning", expectedOutput: "Buenos días" }
        ],
        task: async ({ input }) => translateText(input, 'es'),
        evaluators: [
          async ({ output, expectedOutput }) => ({
            name: "bleu_score",
            value: calculateBleuScore(output, expectedOutput)
          })
        ]
      });

      const result = await langfuse.experiment.run({
        name: "Large Scale Evaluation",
        data: largeBatchOfItems,
        task: expensiveModelCall,
        maxConcurrency: 5, // Process max 5 items simultaneously
        evaluators: [myEvaluator],
        runEvaluators: [
          async ({ itemResults }) => ({
            name: "average_score",
            value: itemResults.reduce((acc, r) => acc + r.evaluations[0].value, 0) / itemResults.length
          })
        ]
      });

    Settings

    Member Visibility

    On This Page

    Constructors
    Accessors
    Methods