Sequential learning, advanced

Configure a Domain for actions that change later states and receive delayed outcomes. Begin with the starting declaration, run the query–execute–feedback loop, then adjust only the settings your task requires. Sequential learning is Adapt-1’s episode-aware policy path for state-changing actions and delayed outcomes. This advanced guide explains how to configure and operate it through the Domain API. When Adapt-1 forms the supported state-dependent action-value and delayed-credit structure from evidence, the Discovery documentation refers to that capability as Sequential Discovery. Declare which states and actions belong to the task and how outcomes become rewards. Choose learning settings for its horizon, feedback density, drift, risk tolerance, and available evidence.

Sequential learning supports authored Domain structure and compatible Discovery paths. The application declares the legal actions, reward semantics, and episode boundaries; supported input structure can form from evidence. The acquisition schedule is separate again: sequential state can form inside a zero-start run, form in a separate trajectory acquisition phase, or continue from a separately acquired checkpoint.

1. When to use sequential learning

Enable sequential learning when all of the following are true:

the system repeatedly chooses among two or more policies or actions;
actions influence later observations;
action quality depends on the current state;
reward may be delayed, sparse, or distributed over an episode;
events can be grouped into episodes and ordered by step;
every action can eventually be linked to a measured outcome.

Examples include control, trading, game play, operations scheduling, treatment sequencing, multi-step remediation, and adaptive workflows. Ordinary contextual policy learning uses outcomes attributed to individual decisions in their decision-time contexts. The application can supply an immediate outcome or compute a return before submitting feedback. Enable the sequential path when Adapt-1 should learn delayed value from ordered transitions. Structured transition learning predicts subsequent observable values. These relationships can coexist in a Domain, with separate targets and output contracts.

Sequential learning is an episode-aware model path inside feedback_policy. A Domain can therefore report feedback_policy and structured_transition as its active subsystems while sequential learning is enabled. When the sequential candidate trains and passes validation, confirm that the serving model is sequential_q_mlp using the model diagnostics below.

Choose zero-start learning, separate acquisition, or both

Setup	Use it when	Main constraint
One zero-start live task	Online adaptation inside that task is the goal, and feedback arrives early enough to affect later choices	One long episode is still one episode and cannot supply episode-held-out validation
Zero-start repeated episodes	Safe exploration and continuing improvement are part of the declared run	Keep one learner scope and use a new episode ID at each real reset; later episodes still use state earned inside the same zero-start run
Separate trajectory acquisition, then freeze	The first held-out action should use learned sequential value, or test-time exploration is unsafe or unavailable	Split acquisition and evaluation episodes before ingestion, preserve action coverage, and stop writes during evaluation
Separate acquisition checkpoint plus live episodes	Prior trajectories form state outside the new run, while the deployed process can drift	Preserve a fixed starting checkpoint so current-run adaptation can be measured separately

The declared run determines the acquisition label: state retained across episodes of a zero-start run was acquired inside that run. Sequential fitting requires the configured minimum_episodes and reserves complete episodes for validation. Confirm the serving model is sequential_q_mlp; policy admission during an individual episode can also update ordinary contextual evidence. A separate trajectory acquisition phase helps only when the state fields, actions, reward, terminal meaning, and useful action-value relationship transfer into the new run. Use a zero-start task-local scope when each independently redrawn task has a different hidden mapping and the current context does not identify that difference. Historical trajectories use the same event and feedback contract as live trajectories. Replay approved records into fresh Domain state, preserve the true episode and step order, wait for asynchronous fitting, and inspect the model report before freezing. See Choose a learning setup for the schedule and scope decision.

2. The learning contract

Each sequential transition must provide this logical tuple:

(episode_id, step, current_context, relation, chosen_policy,
 next_context, step_reward, terminal)

The online loop calls:

POST /domains/{domain_id}/query supplies the current context and returns a selected policy plus a decision_id.
POST /domains/{domain_id}/feedback returns the observed transition and reward.

A decision_id binds feedback to the sealed earlier query. Current runtime context sources are a valid decision_id (decision), a valid target_memory_id (memory), or explicit structured context (request). Sequential readiness is downstream of this policy-admission step. The default field mapping is:

Logical value	Default path in feedback
Episode ID	`metadata.episode_id`
Ordered step	`metadata.step`
Successor state	`values.next_state`
Per-step reward	`values.step_reward`
Terminal flag	`values.terminal`

All five values must be present and correctly typed for a record to be eligible for sequential training:

episode_id: string or other stable identifier;
step: finite number, normally a zero-based integer;
next_state: JSON object;
step_reward: finite number;
terminal: JSON Boolean (true or false).

Scope rule

Keep the same authenticated owner and domain_id when learning should carry across episodes. Use metadata.episode_id to separate environment episodes. On the hosted API, the bearer token determines the effective session; changing the body session_id does not create a new tenant or learner. Use a new Domain for an independent task-state history, or a separate credential for a separate identity. The examples use session_id: ignored for compatibility.

3. Recommended starting declaration

The following declaration uses example parameters for a delayed-reward policy task. Replace the state fields, policies, policy features, and reward limits with values from the application. Numeric settings in the parameter tables are examples; inspect the resolved Domain configuration for the values in use.

{
  "domain_id": "sequential-controller-v1",
  "session_id": "ignored",
  "description": "Select a control policy from state and delayed outcomes.",
  "schema": {
    "entities": ["controller", "workload"],
    "relations": ["controls"],
    "signals": ["queue_depth", "error_rate", "capacity", "step_reward"],
    "event_types": ["state", "feedback"]
  },
  "hypotheses": [
    {
      "name": "conservative controller",
      "relation": "controls",
      "policy": "conservative",
      "policy_features": {
        "intensity": 0.25,
        "risk_class": "low"
      },
      "when": [],
      "predicts": ["stable service with limited intervention"],
      "falsified_by": [],
      "weight": 1.0
    },
    {
      "name": "balanced controller",
      "relation": "controls",
      "policy": "balanced",
      "policy_features": {
        "intensity": 0.55,
        "risk_class": "medium"
      },
      "when": [],
      "predicts": ["balanced recovery and intervention cost"],
      "falsified_by": [],
      "weight": 1.0
    },
    {
      "name": "aggressive controller",
      "relation": "controls",
      "policy": "aggressive",
      "policy_features": {
        "intensity": 0.9,
        "risk_class": "high"
      },
      "when": [],
      "predicts": ["rapid recovery with higher intervention cost"],
      "falsified_by": [],
      "weight": 1.0
    }
  ],
  "learning": {
    "enabled": true,
    "context": {
      "feature_paths": [
        "values.queue_depth",
        "values.error_rate",
        "values.capacity",
        "values.operating_mode"
      ],
      "event_types": ["state"],
      "max_samples": 4096
    },
    "reward": {
      "aggregation": "weighted_mean",
      "components": [
        {
          "field": "values.step_reward",
          "goal": "maximize",
          "min": 0.0,
          "max": 1.0,
          "weight": 1.0,
          "required": true
        }
      ]
    },
    "policy": {
      "decay_half_life_seconds": 0.0,
      "transfer_strength": 0.0,
      "action_transfer_strength": 0.1,
      "min_context_observations": 4,
      "model_max_weight": 0.45,
      "model_ood_threshold": 0.75,
      "ood_confidence_penalty": 0.5,
      "exploration_mode": "auto",
      "exploration_strength": 1.0,
      "minimum_expected_reward": 0.0,
      "minimum_confidence": 0.0,
      "require_credible_dominance": false,
      "dominance_margin": 0.0,
      "abstain_on_ood": false
    },
    "credit_assignment": {
      "mode": "eligibility_trace",
      "discount": 0.95,
      "neutral_reward": 0.5,
      "minimum_weight": 0.01,
      "delay_path": "values.delay_steps"
    },
    "sequential": {
      "enabled": true,
      "episode_path": "metadata.episode_id",
      "step_path": "metadata.step",
      "next_context_path": "values.next_state",
      "reward_path": "values.step_reward",
      "terminal_path": "values.terminal",
      "discount": 0.95,
      "n_step": 6,
      "minimum_episodes": 10,
      "minimum_validation_skill": 0.1,
      "target_sync_interval": 10,
      "conservative_weight": 0.05
    },
    "latent_belief": {
      "enabled": true,
      "max_regimes": 8,
      "novelty_threshold": 0.55,
      "temperature": 0.25,
      "change_point_threshold": 0.55,
      "change_point_patience": 3,
      "historical_regime_weight": 0.05
    },
    "training": {
      "enabled": true,
      "min_samples": 64,
      "retrain_interval": 16,
      "dimensions": 96,
      "epochs": 100,
      "learning_rate": 0.04
    }
  }
}

Create it with:

export ADAPT1_BASE_URL="https://rei-neuroadapt-api.reilabs.org/api/v1"
export ADAPT1_API_KEY="YOUR_API_KEY"

curl -sS -X POST "$ADAPT1_BASE_URL/domains" \
  -H "Authorization: Bearer $ADAPT1_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @domain.json

Configure the learner

4. Define the policy space correctly

Each selectable hypothesis needs:

a name for humans and traces;
the same relation for policies competing in one decision;
a unique, non-null policy identifier;
optional policy_features describing the action itself;
optional when, predicts, and falsified_by clauses for interpretable conditions.

policy identifies an executable action. Its policy_features describe attributes available before execution so the learner can generalize between related actions.Good policy features:

{
  "dose": 0.4,
  "duration_seconds": 30,
  "mode": "pulsed"
}

Poor policy features:

{
  "will_succeed": true,
  "future_reward": 0.9
}

Action selection must resolve to a declared policy. A policy-less induced hypothesis supplies predictive evidence. Keep correct-action labels outside autonomous structure targets used for selection.

5. Choose context features

learning.context.feature_paths determines which state fields are visible to contextual and sequential model training and inference.Include fields that are:

available before the decision;
causally or predictively relevant to action value;
represented consistently at query and feedback time;
stable in name, unit, and type.

Exclude fields that are:

generated after the action;
direct encodings of the answer or reward;
identifiers with no transferable meaning;
timestamps when elapsed time or phase would be more useful;
high-cardinality provenance that encourages memorization.

An empty feature_paths list admits eligible structured leaf fields. Review them for identifiers and logging fields, and exclude outcome leakage. Use explicit paths when the approved inputs must remain fixed.context.event_types filters stored events when a query asks Adapt-1 to infer the latest context. Supply context explicitly when the application has the current observation.context.max_samples bounds retained policy examples. For sequential tasks, size it to retain several representative episodes:

max_samples >= retained_episodes x representative_steps_per_episode

For an example budget of 200-step episodes across 10 retained episodes, 2048 or 4096 provides capacity for the recorded transitions. Replay eviction balances episode groups and preserves failure evidence within the retained sample budget.

6. Define a real reward

Policy learning only updates from measured reward. A successful HTTP response does not imply that the feedback changed the learner.Declarative reward componentsEach component supports:

Field	Meaning
`field`	Dot path inside the feedback envelope, such as `values.profit`
`goal`	`maximize`, `minimize`, or `target`
`min`, `max`	Expected numeric range used to normalize utility to `[0,1]`
`target`	Desired value for a `target` component
`tolerance`	Distance from `target` at which utility reaches zero
`weight`	Relative contribution to weighted or geometric aggregation
`hard`	Force total reward to zero when this component’s utility is zero
`required`	Treat a missing or non-numeric value as a measured hard failure

Aggregation modes:

Mode	Use it when
`weighted_mean`	Objectives can compensate for one another
`geometric_mean`	A very weak component should strongly reduce the total
`minimum`	The weakest resolved objective should define the reward

Example with performance and safety:

{
  "aggregation": "geometric_mean",
  "components": [
    {
      "field": "values.throughput",
      "goal": "maximize",
      "min": 0,
      "max": 100,
      "weight": 2
    },
    {
      "field": "values.error_rate",
      "goal": "minimize",
      "min": 0,
      "max": 0.1,
      "weight": 1,
      "hard": true,
      "required": true
    }
  ]
}

Without components, Adapt-1 recognizes numeric values.reward, values.score, or values.utility; Boolean correct, success, or accepted; and recognized binary outcome labels. Fields such as error_distance have no implicit reward meaning. Either declare them as components or compute a normalized reward externally.Reward scale and neutral rewardKeep rewards in [0,1]. The default neutral point is 0.5:

below 0.5 is disadvantageous;
0.5 is neutral or unresolved;
above 0.5 is advantageous.

For a naturally signed reward, define normalization explicitly. For example, map profit from [-100,100] with min: -100, max: 100, and goal: maximize.Set component bounds explicitly and inspect the reward values actually admitted. Sequential learning also depends on the accumulated return: repeated extreme rewards can saturate bounded return targets and flatten action-value differences. If estimates cluster at 0 or 1, inspect normalization, neutral reward, discount, and n_step before changing selection gates. A tied query alone does not identify the cause.Keep contextual reward and sequential reward_path meanings explicit. If several recognized reward fields are present, follow the feedback field precedence and avoid conflicting signals.Do not invent a success or failure label for an unknown outcome. Use an outcome value accepted by the current feedback schema and provide the numeric field required by the reward declaration. Do not rely on undocumented outcome labels; the live validator is authoritative for accepted enum values.

7. Configure sequential return learning

Sequential parameters

Parameter	Example value	Effect	Tuning guidance
`enabled`	`false`	Enables episode-aware return learning	Must be `true` for sequential Q learning
`episode_path`	`metadata.episode_id`	Groups transitions into episodes	Keep stable across one episode and distinct across episodes
`step_path`	`metadata.step`	Orders transitions within an episode	Use monotonic, unique step values
`next_context_path`	`values.next_state`	Supplies the observed successor state	Must resolve to an object on every eligible feedback row
`reward_path`	`values.step_reward`	Supplies the sequential reward	Must resolve to a finite number
`terminal_path`	`values.terminal`	Marks episode completion	Send a real Boolean on every row
`discount`	`0.95`	Controls how much future value affects earlier actions	Lower for short horizons; higher for long delayed effects
`n_step`	`3`	Number of observed rewards used before bootstrapping	Increase for delayed rewards; reduce for dense/noisy rewards
`minimum_episodes`	`8`	Minimum distinct episodes before sequential fitting	Preserve enough episodes for training and held-out validation; increase when episodes vary substantially
`minimum_validation_skill`	`0.25`	Required held-out improvement over a constant-return baseline	Lower cautiously for noisy data; never use it to force a failing model into selection
`target_sync_interval`	`10`	Epoch interval for copying online weights to the target network	Increase for a slower, more stable target; reduce for faster tracking
`conservative_weight`	`0.05`	Penalizes unsupported action values	Increase when action coverage is sparse or overestimation is costly

The learner reserves the last portion of ordered episode IDs for validation. Installation requires validation skill to meet the declared threshold, so retained data must include complete episodes for both fitting and validation.Picking discountUse the effective horizon as a guide:

effective horizon is approximately 1 / (1 - discount)

Approximate examples:

`discount`	Effective horizon
`0.80`	5 steps
`0.90`	10 steps
`0.95`	20 steps
`0.98`	50 steps
`0.99`	100 steps

Very high discount values can propagate noise and make policies hard to distinguish. Start from the shortest horizon that can still reach the delayed consequence of interest.Picking n_step

Dense reward every step: start at 1 to 3.
Reward after a short action chain: start at 4 to 8.
Sparse terminal reward: start at 8 to 20, with eligibility traces enabled.
Highly stochastic outcomes: use a smaller value and more episodes.

n_step controls the observed return window before the model bootstraps from the successor state. For example, n_step: 3 sets a three-reward window before bootstrapping; the successor value can include later consequences. Planning depth is configured separately in a transition-planning workflow. See Decisions beyond the next move.

8. Configure delayed credit

credit_assignment.mode controls synchronous reward attribution in the online policy state. Sequential Q learning separately consumes the episode transitions and returns.

Parameter	Example value	Effect
`mode`	`none`	Use `eligibility_trace` to propagate delayed non-neutral outcomes
`discount`	`0.9`	Exponential decay applied per delay step
`neutral_reward`	`0.5`	Baseline around which positive and negative credit moves
`minimum_weight`	`0.0`	Stops backward propagation after the discounted weight falls below this value
`delay_path`	`values.delay_steps`	Optional explicit delay used to adjust the current feedback reward

With eligibility traces enabled, a later non-neutral step_reward can update earlier neutral transitions in the same episode. Propagation stops at a prior non-neutral signal, a terminal boundary, an unchanged transition, or the minimum weight cutoff.Use mode: none when rewards are immediate and accurately assigned. Use eligibility_trace when the effect arrives later than the responsible action.Do not use a large minimum_weight for long sparse-reward episodes. With discount: 0.95, a cutoff of 0.05 reaches roughly 58 steps; a cutoff of 0.01 reaches roughly 90 steps.

9. Configure exploration and selection safety

Exploration modes

Mode	Behavior	Best use
`exploit`	Select the highest current expected return	Frozen evaluation and stable deployment
`ucb`	Adds uncertainty-scaled optimism	Broad, controlled exploration
`thompson`	Samples from each policy posterior	Stochastic exploration proportional to uncertainty
`auto`	Chooses among exploit, UCB, and Thompson from confidence and bound-transition status	General online operation

Exploration requires allow_exploration: true in the query and uses the mode configured by the Domain. A query can temporarily override the mode with selection_mode.Record the returned mode when inspecting auto; its choice can remain unchanged across successive decisions. Posterior sampling can be reproducible for a sealed decision. Compare the context, admitted evidence, and returned diagnostics when investigating repeated selections.exploration_strength scales UCB uncertainty. 0 removes the UCB bonus. Values near 0.5 are cautious, 1.0 is a reasonable start, and values above 1.0 deliberately favor underexplored actions.Selection gates

Parameter	Example value	Effect
`minimum_expected_reward`	`0.0`	Abstain if the best candidate is below this expected reward
`minimum_confidence`	`0.0`	Abstain if contextual confidence is too low
`require_credible_dominance`	`false`	Require the winner’s credible interval to dominate competitors
`dominance_margin`	`0.0`	Required separation when credible dominance is enabled
`abstain_on_ood`	`false`	Abstain when the context is outside learned support

Set selection gates according to the permitted deployment risk. Use held-out operational data to choose thresholds after collecting initial coverage. The client must always handle status: "abstained"; abstention is a valid safety output.

10. Configure transfer and nonstationarity

Policy transfer

Parameter	Example value	Meaning
`transfer_strength`	`0.15`	Weight of same-policy evidence learned under another relation
`action_transfer_strength`	`0.05`	Weight of evidence from different policies with similar `policy_features`
`min_context_observations`	`4`	Local support target used by confidence and calibration machinery
`model_max_weight`	`0.45`	Maximum contribution of the asynchronously trained model
`model_ood_threshold`	`0.75`	Novelty distance at which model contribution falls to zero
`ood_confidence_penalty`	`0.5`	Confidence reduction under out-of-distribution context

Set both transfer strengths to 0 when actions or relations have unrelated semantics. Increase action_transfer_strength only when policy features have stable physical or operational meaning. Transfer never crosses owner, Domain, or session scope.Decaydecay_half_life_seconds exponentially reduces the influence of older feedback. 0 disables time decay.Use decay for regime changes, market drift, changing users, or equipment aging. Choose a half-life in application time that smooths ordinary noise and allows obsolete evidence to decay.Examples:

stable physical process: 0 or several months;
weekly operational drift: several days to weeks;
intraday regime changes: tens of minutes to hours;
controlled reversal test: long enough to retain evidence, short enough for new evidence to dominate.

11. Configure latent belief

Latent belief separates recurring or changing context regimes without requiring the client to provide a regime label.

Parameter	Example value	Effect
`enabled`	`true`	Enables latent regime inference
`max_regimes`	`8`	Maximum regimes retained per relation
`novelty_threshold`	`0.55`	Context distance needed to create a new regime
`temperature`	`0.25`	Softness of regime probability assignment
`change_point_threshold`	`0.55`	Residual threshold for suspected behavior change
`change_point_patience`	`3`	Consecutive high residuals required for a change point
`historical_regime_weight`	`0.05`	Residual influence of evidence from non-active regimes

Tuning effects:

Lower novelty_threshold: creates regimes more readily; useful for distinct modes, risky with noisy features.
Higher novelty_threshold: merges more contexts into one regime.
Lower temperature: sharper regime assignments.
Higher temperature: blends evidence across nearby regimes.
Lower change_point_patience: reacts faster and risks false changes.
Higher historical_regime_weight: gives evidence from earlier regimes more influence.

Tune latent belief after verifying feature scaling and reward semantics, including episode linkage.

12. Configure asynchronous model training

A fresh Domain starts these policy learners without task-trained state. Eligible feedback updates the online policy posterior synchronously and triggers background candidate fitting during operation when the configured training gates are met. Queries use the available policy state while a candidate is fitted and validated.

Parameter	Example value	Effect	Guidance
`enabled`	`true`	Enables asynchronous model fitting	Disable only when posterior-only behavior is required
`min_samples`	`24`	First sample count at which training may start	Use `64` to `256` for noisier sequential tasks
`retrain_interval`	`8`	New policy samples required after the last completed fit	Increase to reduce training churn; decrease for fast adaptation
`dimensions`	`96`	Capacity control for the contextual model	Sequential Q input capacity is derived from the declared feature schema
`epochs`	`100`	Optimization epochs per fit	Increase when held-out validation indicates underfitting
`learning_rate`	`0.04`	Optimizer step size	Reduce if validation is unstable; increase cautiously if learning is consistently slow

retrain_interval controls how many newly accepted samples are required after a completed fit. Choose an interval for the acceptable compute load and adaptation delay. Example settings:

rapid drift, cheap fitting: 8 to 20;
stable task, moderate load: 32 to 100;
expensive fitting or very large streams: 100 or more.

Training readiness also depends on minimum_episodes. A fit triggered before enough episodes exist may train the ordinary contextual candidate; once enough valid episodes accumulate, the sequential candidate can be trained and validated.

13. Execute the online loop

Step 1: query before acting

curl -sS -X POST \
  "$ADAPT1_BASE_URL/domains/sequential-controller-v1/query" \
  -H "Authorization: Bearer $ADAPT1_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "ignored",
    "question": "Select the controller for the current state.",
    "relation": "controls",
    "context": {
      "event_type": "state",
      "values": {
        "queue_depth": 71,
        "error_rate": 0.04,
        "capacity": 0.62,
        "operating_mode": "degraded"
      }
    },
    "allow_exploration": true,
    "selection_mode": "auto"
  }'

Persist these response fields:

decision_id;
selection.status;
selection.selected_policy;
the exact action actually executed;
the pre-action context.

If the application overrides Adapt-1’s selection, send feedback for the policy that was actually executed. Do not credit an action that was only proposed.

Step 2: observe the transition

After executing the selected policy, collect:

the successor state;
the reward components;
whether the episode ended;
the episode ID and step;
any measured delay.

Step 3: write decision-linked feedback

curl -sS -X POST \
  "$ADAPT1_BASE_URL/domains/sequential-controller-v1/feedback" \
  -H "Authorization: Bearer $ADAPT1_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "ignored",
    "outcome": "success",
    "feedback_kind": "execution",
    "decision_id": "DECISION_ID_FROM_QUERY",
    "relation": "controls",
    "policy": "SELECTED_POLICY_FROM_QUERY",
    "values": {
      "step_reward": 0.65,
      "next_state": {
        "queue_depth": 54,
        "error_rate": 0.025,
        "capacity": 0.61,
        "operating_mode": "recovering"
      },
      "terminal": false,
      "delay_steps": 1
    },
    "metadata": {
      "episode_id": "episode-0001",
      "step": 0
    }
  }'

Repeat query, action, and feedback until the terminal transition. The terminal row still needs a valid next_state object and must send "terminal": true. decision_id binds the query → action → outcome loop to sealed pre-action context. Current runtime policy admission can also recover context from target_memory_id or from explicit structured context. In every case, send the executed relation and policy explicitly and verify that credit_assignment.contextual_learning_applied is true. Current admission checks require the relation field to be present and do not validate its value against the bound decision. Send the real relation from the Domain. A successful feedback response without relation can store the record while leaving feedback_policy.sample_count unchanged.

14. Batch feedback without losing sequence semantics

POST /domains/{domain_id}/batch accepts ordered event and feedback operations. Every feedback operation retains its own decision_id, metadata.episode_id, metadata.step, successor state, reward, and terminal flag. Batch feedback when the corresponding decisions and outcomes have already occurred. Preserve each transition identity and order. Continue querying between actions when a new action depends on the previous result. Minimal shape:

{
  "session_id": "ignored",
  "operations": [
    {
      "operation": "feedback",
      "ref": "feedback-episode-1-step-0",
      "outcome": "success",
      "feedback_kind": "execution",
      "decision_id": "decision-step-0",
      "relation": "controls",
      "policy": "balanced",
      "values": {
        "step_reward": 0.5,
        "next_state": {"queue_depth": 62, "error_rate": 0.03, "capacity": 0.62, "operating_mode": "degraded"},
        "terminal": false,
        "delay_steps": 0
      },
      "metadata": {"episode_id": "episode-0001", "step": 0}
    },
    {
      "operation": "feedback",
      "ref": "feedback-episode-1-step-1",
      "outcome": "success",
      "feedback_kind": "execution",
      "decision_id": "decision-step-1",
      "relation": "controls",
      "policy": "balanced",
      "values": {
        "step_reward": 0.9,
        "next_state": {"queue_depth": 30, "error_rate": 0.01, "capacity": 0.65, "operating_mode": "normal"},
        "terminal": true,
        "delay_steps": 0
      },
      "metadata": {"episode_id": "episode-0001", "step": 1}
    }
  ]
}

15. Read the diagnostics

Request a normal Domain query and inspect:

Output	What it tells you
`selection.status`	`selected` or `abstained`
`selection.reason`	Why no policy was selected or which gate applied
`selection.selected_policy`	Executable action identity
`ranked_hypotheses[].contextual_policy.expected_reward`	Fused expected immediate/contextual reward
`ranked_hypotheses[].contextual_policy.selection_expected_reward`	Value actually preserved for policy ordering, including sequential return
`ranked_hypotheses[].contextual_policy.sequential_expected_reward`	Sequential model estimate when installed
`ranked_hypotheses[].contextual_policy.model_weight`	Effective trained-model contribution
`ranked_hypotheses[].contextual_policy.out_of_distribution`	Whether context novelty exceeded support
`ranked_hypotheses[].contextual_policy.effective_observations`	Weighted evidence supporting the estimate
`latent_belief`	Current regime probabilities and novelty
`learning_state.subsystems.feedback_policy.sample_count`	Retained eligible policy-learning observations, bounded by `context.max_samples`
`learning_state.subsystems.feedback_policy.model.status`	Fitted policy-model component’s training or retention state, including `waiting`, `running`, `trained`, `installed`, `retained`, or `failed`
`learning_state.subsystems.feedback_policy.model.current_model_type`	Serving fitted estimator within `feedback_policy`, when reported, including after candidate rejection
`learning_state.subsystems.feedback_policy.model.report.model_type`	Model described by the latest fit report; check whether it was installed or the incumbent retained
`learning_state.subsystems.feedback_policy.model.report.validation_skill`	Reported fit’s held-out improvement over the constant-return baseline

The top-level learning_state.sample_count is an aggregate across active learning subsystems. Use the feedback-policy subsystem count when diagnosing sequential policy learning. Diagnose sequential fitting through learning_state.subsystems.feedback_policy. Its admitted feedback samples supply the episode and training gates for sequential_q_mlp. Check this admission before investigating training thresholds.

Serving models, candidate reports, and fallback

Within learning_state.subsystems.feedback_policy.model, report describes a fit. After a candidate is rejected, it can describe that candidate while current_model_type and current_validation_skill identify the retained serving model. A response with status: retained and installed: false can therefore still have an active model. After a successful installation, the report describes the installed fit. trained records a completed fit; inspect installation and retention fields together.The model identifiers here describe fitted estimators available to the configured feedback_policy inference path. Contextual candidates can report extra_trees or mlp_v2; sequential_q_mlp identifies the sequential return path. Candidate validation contributes to selecting the serving estimator. Query model_weight reports that estimator’s contribution for the current context. The Mechanisms page describes how model fitting operates alongside learned task structure, retained evidence, and regime state. report.sample_count describes the fit’s training data and can lag the current learner count.The reason fields describe different decisions:

model.reason: why fitting retained the incumbent or rejected a candidate, such as candidate_failed_validation or candidate_objective_mismatch;
model.report.selection_reason: model selection, such as robust_model_within_validation_tolerance;
query selection.reason: action selection or abstention.

In contextual-policy diagnostics, model_distribution.status: latent_fallback with model_support_unavailable in model_distribution.reasons[] means trained-model support was unavailable for that estimate. Check model_weight and OOD support alongside it. These diagnostics distinguish a completed fit from useful model contribution on the current query.

Verify sequential learning through these diagnostics:

accepted feedback updates the feedback-policy subsystem; inspect its sample count and learner version, allowing for the retained-sample limit;
multiple valid episode IDs are retained;
fitting completes and a validated model is installed or retained;
the serving model type is sequential_q_mlp;
the serving model’s validation skill meets the configured threshold;
sequential expected rewards differ across policies or contexts and contribute to selection;
evaluate frozen exploit decisions on held-out episodes to measure the effect on task performance.

16. Freeze evaluation correctly

To evaluate a frozen learned state:

train only on the training partition;
wait for asynchronous training to complete;
stop all /feedback writes;
query held-out episodes with allow_exploration: false;
use selection_mode: "exploit";
keep the same authenticated owner and domain_id so the trained state remains available;
use unseen episode IDs, and preferably unseen scenarios or entities;
record abstentions in the Adapt-1 score and score application fallbacks separately.

Frozen query example:

{
  "session_id": "ignored",
  "question": "Select the controller for this held-out state.",
  "relation": "controls",
  "context": {
    "event_type": "state",
    "values": {
      "queue_depth": 83,
      "error_rate": 0.052,
      "capacity": 0.58,
      "operating_mode": "degraded"
    }
  },
  "allow_exploration": false,
  "selection_mode": "exploit"
}

Report at least return, policy accuracy if ground truth exists, abstention coverage, selective performance, episode variance, and learning curves by episode. Use multiple seeds or independent streams for stochastic environments.

Advanced configuration

17. Tuning profiles

These profiles provide example parameters for evaluation on the application task.Sparse delayed reward

{
  "context": {"max_samples": 4096},
  "credit_assignment": {
    "mode": "eligibility_trace",
    "discount": 0.97,
    "neutral_reward": 0.5,
    "minimum_weight": 0.005
  },
  "sequential": {
    "discount": 0.98,
    "n_step": 12,
    "minimum_episodes": 12,
    "conservative_weight": 0.08
  },
  "training": {"min_samples": 128, "retrain_interval": 32}
}

Use when most steps are neutral and terminal outcomes carry the useful signal.Dense short-horizon control

{
  "credit_assignment": {"mode": "none"},
  "sequential": {
    "discount": 0.9,
    "n_step": 3,
    "minimum_episodes": 8,
    "conservative_weight": 0.03
  },
  "training": {"min_samples": 64, "retrain_interval": 16}
}

Nonstationary or reversal-prone task

{
  "policy": {
    "decay_half_life_seconds": 86400,
    "transfer_strength": 0.0,
    "action_transfer_strength": 0.05
  },
  "latent_belief": {
    "enabled": true,
    "max_regimes": 12,
    "novelty_threshold": 0.45,
    "change_point_threshold": 0.4,
    "change_point_patience": 2,
    "historical_regime_weight": 0.1
  },
  "training": {"retrain_interval": 8}
}

Set the decay half-life in real application time. 86400 is one day and is only an example.High-stakes conservative deployment

{
  "policy": {
    "exploration_mode": "exploit",
    "minimum_expected_reward": 0.6,
    "minimum_confidence": 0.4,
    "require_credible_dominance": true,
    "dominance_margin": 0.05,
    "abstain_on_ood": true
  },
  "sequential": {
    "minimum_validation_skill": 0.25,
    "conservative_weight": 0.12
  }
}

Collect exploration data in a safe environment before enabling these deployment gates.Many structurally similar actionsUse informative policy_features, set action_transfer_strength around 0.1 to 0.25, and increase conservative_weight when many actions remain unsupported. Compare against action_transfer_strength: 0 to measure the effect on held-out actions and check for interference between distinct actions.

18. Advanced bound-transition projection

learning.sequential.bound_transition is an optional non-neural projection for tasks with explicitly bound state objects and action roles. It can estimate how a declared transition action changes a bounded objective and blend that estimate into policy scoring. Most sequential tasks should leave it disabled.It is appropriate when:

one context contains multiple indexed entities or objects;
a policy selects which entity is the current state and optionally which is the action source;
action roles can be declared from policy features;
the objective has known numeric bounds;
state transformations can transfer between structurally similar bindings.

Parameter	Meaning
`enabled`	Enables bound-transition projection
`state_selector_feature`	Policy-feature path whose value selects the current state object
`state_features`	Map from logical feature name to context path template; `{selector}` is substituted
`state_bounds`	Optional `[min,max]` validation bounds by logical state feature
`action_selector_feature`	Optional policy-feature path selecting an action/source object
`action_features`	Map from logical action feature to templated context path
`signature_features`	Policy-feature paths that must match for direct transition evidence
`objective_feature`	Logical state feature to maximize after normalization
`episode_start_path`	Optional context Boolean that resets episode-local binding
`objective_min`, `objective_max`	Numeric normalization limits
`transition_values`	Policy-feature path to allowed values identifying transition actions
`terminal_values`	Policy-feature path to allowed values identifying terminal actions
`minimum_support`	Required matching transitions before prediction
`model_weight`	Blend weight of the bound projection in policy scoring
`novelty_bonus`	Exploration value assigned to unsupported transitions
`transition_cost`	Utility subtracted from predicted transition actions

Enable bound-transition projection when the application can declare selectors, state features, action roles, and objective semantics independently of evaluation labels.

19. Common failure modes

`sample_count` stays at zero

First distinguish storage from learner admission. A successful feedback response can store the record while leaving feedback_policy.sample_count unchanged. Check:

learning.enabled is true;
feedback includes a real measured reward;
relation is present in the feedback request;
the executed policy is present and is one of the declared hypotheses;
decision-time context is recoverable from a valid decision_id, a valid target_memory_id, or explicit structured context.

Then inspect credit_assignment.contextual_learning_applied. If it is false, the record did not enter contextual policy learning even if the HTTP request succeeded. Current responses report the context route under credit_assignment.context_source as decision, memory, or request. An unrecognized outcome label does not update policy learning unless a numeric, Boolean, or declarative reward is also present.

Feedback accumulates without an installed sequential model

Check every row for all five configured sequential paths and types. Then check:

at least minimum_episodes distinct episode IDs exist;
replay capacity retains enough rows from those episodes;
training has reached min_samples;
Torch or the configured training service is available;
asynchronous fitting has completed;
validation skill meets minimum_validation_skill.

Every decision abstains at the start

Use allow_exploration: true during data collection. With no policy evidence and exploration disabled, abstention is expected. Keep strict confidence, dominance, and OOD gates disabled until there is enough coverage.

Decisions return `selected_policy: null`

Only declared hypotheses with non-null policies are executable. Predictive or induced hypotheses without a policy cannot be selected as actions. Verify that all competing policy hypotheses share the requested relation and have unique policies.

Learning becomes worse across episodes

Check for:

reward direction or normalization errors;
incorrect decision-to-feedback linkage;
outcome leakage in context features;
reused episode IDs or non-monotonic steps;
too-small replay capacity;
excessive transfer between unrelated actions;
too-high discount or n_step for noisy rewards;
validation episodes drawn from a different regime due to lexicographic episode naming;
exploration still enabled during evaluation;
client fallbacks being scored as if Adapt-1 selected them.

Policy ordering stays flat after training

Inspect validation_skill, sequential_expected_reward, model weight, and OOD status. Common causes are constant rewards, action-independent features, identical policy features, low action coverage, a feature path mismatch between current and successor state, or OOD downweighting.

20. Building the learner for a new use case

Use this sequence:

Define the decision. Write one sentence: “Given state X, choose one policy from Y to maximize measured outcome Z over horizon H.”
Declare policies. Give every executable action a unique policy and observable policy features.
Declare pre-action state. Select only fields known before acting.
Define reward mathematically. Normalize every objective to [0,1]; specify tradeoffs and hard constraints.
Define episode boundaries. Establish stable episode IDs, ordered steps, successor state, and terminal semantics.
Estimate the horizon. Pick the shortest discount and n_step that reach the relevant consequences.
Size replay. Retain enough transitions from enough complete and diverse episodes.
Choose exploration. Explore in a safe environment; exploit or abstain in high-risk deployment.
Set validation gates. Require positive held-out skill and keep conservative penalties when action coverage is incomplete.
Instrument the loop. Persist decisions and verify that every executed action receives correctly linked feedback.
Run ablations. Compare sequential enabled/disabled, transfer enabled/disabled, and latent belief enabled/disabled.
Freeze and evaluate. Use unseen episodes with no feedback writes and no exploration.

21. Pre-deployment checklist

Sequential Discovery

Start with the Discovery behavior, public boundary, and minimal contract.

Discovery overview

Review Transition, Structure, and Sequential Discovery together.

Get started

Concepts

Guides

Discovery

Operations

API reference

1. When to use sequential learning

Choose zero-start learning, separate acquisition, or both

2. The learning contract

Scope rule

3. Recommended starting declaration

Configure the learner

13. Execute the online loop

Step 1: query before acting

Step 2: observe the transition

Step 3: write decision-linked feedback

14. Batch feedback without losing sequence semantics

15. Read the diagnostics

16. Freeze evaluation correctly

Advanced configuration

19. Common failure modes

`sample_count` stays at zero

Feedback accumulates without an installed sequential model

Every decision abstains at the start

Decisions return `selected_policy: null`

Learning becomes worse across episodes

Policy ordering stays flat after training

Sequential Discovery

Discovery overview

​1. When to use sequential learning

​Choose zero-start learning, separate acquisition, or both

​2. The learning contract

​Scope rule

​3. Recommended starting declaration

​Configure the learner

​13. Execute the online loop

​Step 1: query before acting

​Step 2: observe the transition

​Step 3: write decision-linked feedback

​14. Batch feedback without losing sequence semantics

​15. Read the diagnostics

​16. Freeze evaluation correctly

​Advanced configuration

​19. Common failure modes

​sample_count stays at zero

​Feedback accumulates without an installed sequential model

​Every decision abstains at the start

​Decisions return selected_policy: null

​Learning becomes worse across episodes

​Policy ordering stays flat after training

Sequential Discovery

Discovery overview

1. When to use sequential learning

Choose zero-start learning, separate acquisition, or both

2. The learning contract

Scope rule

3. Recommended starting declaration

Configure the learner

13. Execute the online loop

Step 1: query before acting

Step 2: observe the transition

Step 3: write decision-linked feedback

14. Batch feedback without losing sequence semantics

15. Read the diagnostics

16. Freeze evaluation correctly

Advanced configuration

19. Common failure modes

`sample_count` stays at zero

Feedback accumulates without an installed sequential model

Every decision abstains at the start

Decisions return `selected_policy: null`

Learning becomes worse across episodes

Policy ordering stays flat after training