One or more errors occurred. (Schema mismatch for score column 'Score': expected vector of two or more items of type Single, got Vector<Single, 1> Arg_ParamName_Name)

[rzechu]

System Information (please complete the following information):

• OS & Version: Windows 11
• ML.NET Version: ML.NET v0.20.0
• .NET Version: NET 6.0

Describe the bug
One or more errors occurred. (Schema mismatch for score column 'Score': expected vector of two or more items of type Single, got Vector<Single, 1> Arg_ParamName_Name)
on AutoML Multiclasstraining

SELECT  
CAST(DecimalColumn1 as real) as [DecimalColumn1], 
CAST(IntColumn2 as real) as [IntColumn2], 
CAST(IntColumn3 as real) as [IntColumn3], 
CAST(IntColumn4 as real) as [IntColumn4], 
cast([LongStringColumn1] as varchar(2000)) as [LongStringColumn1],  /*nvarchar(max) */
[Name], /*nvarchar(128) */
[ShortStringColumn3] as [ShortStringColumn3], /*nvarchar(128) */
cast([LongStringColumn3] as varchar(2000)) as [LabelColumn]  /*nvarchar(max) */
FROM Documents 

To Reproduce
Steps to reproduce the behavior:

1. MulticlassClassification
2. Loading data from MSSQL Database

var loader = MLContext.Data.CreateDatabaseLoader(columns.ToArray());
var dbSource = new DatabaseSource(SqlClientFactory.Instance, connectionString, sqlQuery);
var iDataView = loader.Load(dbSource);

columns
    [0]: "Name=DecimalColumn1      Type=Single"
    [1]: "Name=IntColumn2               Type=Single"
    [2]: "Name=IntColumn3               Type=Single"
    [3]: "Name=IntColumn4               Type=Single"
    [4]: "Name=LongStringColumn1  Type=String"
    [5]: "Name=Name                        Type=String"
    [6]: "Name=ShortStringColumn3 Type=String"
    [7]: "Name=LabelColumn             Type=String"

iDataView
    [0]: "Index=0 Name=DecimalColumn1     Type=Single IsHidden=False"
    [1]: "Index=1 Name=IntColumn2               Type=Single IsHidden=False"
    [2]: "Index=2 Name=IntColumn3               Type=Single IsHidden=False"
    [3]: "Index=3 Name=IntColumn4               Type=Single IsHidden=False"
    [4]: "Index=4 Name=LongStringColumn1  Type=String IsHidden=False"
    [5]: "Index=5 Name=Name                         Type=String IsHidden=False"
    [6]: "Index=6 Name=ShortStringColumn3 Type=String IsHidden=False"
    [7]: "Index=7 Name=LabelColumn             Type=String IsHidden=False"

iDataView.Preview()

var experiment = MLContext.Auto().CreateMulticlassClassificationExperiment(multiClassExperimentSettings);
var experimentResult = experiment.Execute(trainData: iDataView, labelColumnName: runExperimentDto.LabelColumnName, progressHandler: progressHandler);

Error:

Schema mismatch for score column 'Score': expected vector of two or more items of type Single, got Vector<Single, 1> (Parameter 'schema')
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.ML.AutoML.AutoMLExperiment.Run()
   at Microsoft.ML.AutoML.MulticlassClassificationExperiment.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
   at AI.AutoML.Trainers.MulticlassClassification.RunExperiment(IDataView trainingDataView, RunExperimentDto runExperimentDto) in C:\.....\_myapp.cs:line 63

InnerException

Schema mismatch for score column 'Score': expected vector of two or more items of type Single, got Vector<Single, 1> (Parameter 'schema')

   at Microsoft.ML.Data.MulticlassClassificationEvaluator.CheckScoreAndLabelTypes(RoleMappedSchema schema)
   at Microsoft.ML.Data.EvaluatorBase`1.CheckColumnTypes(RoleMappedSchema schema)
   at Microsoft.ML.Data.EvaluatorBase`1.Microsoft.ML.Data.IEvaluator.Evaluate(RoleMappedData data)
   at Microsoft.ML.Data.MulticlassClassificationEvaluator.Evaluate(IDataView data, String label, String score, String predictedLabel)
   at Microsoft.ML.MulticlassClassificationCatalog.<>c__DisplayClass6_0.<CrossValidate>b__0(CrossValidationResult x)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
   at Microsoft.ML.AutoML.MulticlassClassificationRunner.Run(TrialSettings settings)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__26.MoveNext()

Expected behavior
This case should work (I am using same scenario for many other columns and only few cases gives this error - propably related to nvarchar(max) columns or some kind of mix columns
I am many times using 2 varchars + 2 ints as source with int/varchar output and work without any doubts
Or should be understandable workaround

[LittleLittleCloud]

How many distinct values do you have for LabelName column? Based on exception message it seems that your label column only have one distinct value?

[rzechu]

How many distinct values do you have for LabelName column? Based on exception message it seems that your label column only have one distinct value?

3 values 2 distinct (a, b, a)
https://user-images.githubusercontent.com/45091660/219575314-5246a738-d1c2-4e42-9c2a-8f90225fdfbd.png

[LittleLittleCloud]

@rzechu
Is it possible for you to export part of your SQL database as csv and give it to us? We want to try reproducing the issue you have on our side

And is your dataset imbalance in labels? Could it be possible that one label has too less instance and it only appears in test dataset rather than train dataset?

[rzechu]

@rzechu Is it possible for you to export part of your SQL database as csv and give it to us? We want to try reproducing the issue you have on our side

CREATE TABLE [dbo].[AI](
	[DecimalColumn1] [decimal](20, 4) NULL,
	[IntColumn2] [int] NULL,
	[IntColumn3] [int] NULL,
	[IntColumn4] [int] NULL,
	[LongStringColumn1] [nvarchar](max) NULL,
	[Name] [nvarchar](255) NULL,
	[ShortStringColumn3] [nvarchar](128) NULL,
	[LabelColumn] [nvarchar](max) NULL
) 
GO
INSERT [dbo].[Ai] ([DecimalColumn1], [IntColumn2], [IntColumn3], [IntColumn4], [LongStringColumn1], [Name], [ShortStringColumn3], [LabelColumn]) VALUES (NULL, 73, 0, NULL, N'Paryż', N'Delegacja zagraniczna | Administrator Systemu', NULL, N'Konferencja')
GO
INSERT [dbo].[Ai] ([DecimalColumn1], [IntColumn2], [IntColumn3], [IntColumn4], [LongStringColumn1], [Name], [ShortStringColumn3], [LabelColumn]) VALUES (NULL, 8, 0, 1583, N'Warszawa', N'Delegacja krajowa | Alicja Konieczny', NULL, N'Szkolenie')
GO
INSERT [dbo].[Ai] ([DecimalColumn1], [IntColumn2], [IntColumn3], [IntColumn4], [LongStringColumn1], [Name], [ShortStringColumn3], [LabelColumn]) VALUES (NULL, 73, 0, 1583, N'Warszawa', N'Delegacja krajowa | Administrator System', NULL, N'Konferencja')
GO

SELECT 
CAST(DecimalColumn1 as real) as [DecimalColumn1], 
CAST(IntColumn2 as real) as [IntColumn2], 
CAST(IntColumn3 as real) as [IntColumn3], 
CAST(IntColumn4 as real) as [IntColumn4], 
[LongStringColumn1], 
[Name] as [Name], 
[ShortStringColumn3] as [ShortStringColumn3], 
[LabelColumn] as [LabelColumn] 
FROM IAi

Its just 3 row example - no more rows
Not diving data into datasets I am using AutoML and its done behind the scene as I know

var connectionString = ConnectionString;
var columns = new SchemaTools().MapDataTableToDataBaseLoaderColumns(sqlQuery, connectionString);
var loader = MLContext.Data.CreateDatabaseLoader(columns.ToArray());
var dbSource = new DatabaseSource(SqlClientFactory.Instance, connectionString, sqlQuery);
var iDataView = loader.Load(dbSource);
var experiment = MLContext.Auto().CreateMulticlassClassificationExperiment(multiClassExperimentSettings);
var experimentResult = experiment.Execute(trainData: iDataView, labelColumnName: runExperimentDto.LabelColumnName, progressHandler: progressHandler);

using Microsoft.ML.Data;
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Linq;

public class SchemaTools
{
    private readonly static Dictionary<Type, DbType> _TypeToSqlDataType = new Dictionary<Type, DbType>()
    {
        { typeof(String), DbType.String },
        { typeof(Int32), DbType.Int32 },
        { typeof(Decimal),DbType.Single },
        { typeof(Double), DbType.Single },
        { typeof(Single), DbType.Single },
        { typeof(DateTime), DbType.DateTime },
        { typeof(Byte[]), DbType.Binary },
    };

    private static DataTable GetDataBaseSchemaColumns(string sqlQuery, string connectionString)
    {
        using (var connection = new SqlConnection(connectionString))
        using (var command = connection.CreateCommand())
        {
            command.CommandText = sqlQuery;
            connection.Open();

            return command.ExecuteReader(CommandBehavior.SchemaOnly)
                .GetSchemaTable();
        }
    }

    public IList<DatabaseLoader.Column> MapDataTableToDataBaseLoaderColumns(string sqlQuery, string connectionString)
    {
        var dataSchema = GetDataBaseSchemaColumns(sqlQuery, connectionString);

        return dataSchema.Rows.Cast<DataRow>()
                .Select(row => MapDataTypeToSqlDataType(row))
                .Where(w => w.Type != DbType.Binary)
                .ToList();
    }

    private DatabaseLoader.Column MapDataTypeToSqlDataType(DataRow dataRow)
    {
        DbType dbType;
        var dataType = dataRow.Field<Type>("DataType");
        var columnName = dataRow.Field<string>("ColumnName");
        if (_TypeToSqlDataType.TryGetValue(dataType, out dbType))
        {
            return new DatabaseLoader.Column()
            {
                Type = dbType,
                Name = columnName
            };
        }
       
        throw new NotImplementedException($"{dataType} can't be mapped to SqlDbType");
    }

[LittleLittleCloud]

If there's only three rows of dataset, it's possible that after divided the training set only contains one label value and validation set contains another. (training set contains row 0 and row 2, validation set contains row 1)

To validate that assumption, can you try setting validation dataset using the same IDataView as training dataset which avoids the train-validation split by AutoML API.

[rzechu]

If there's only three rows of dataset, it's possible that after divided the training set only contains one label value and validation set contains another. (training set contains row 0 and row 2, validation set contains row 1)

To validate that assumption, can you try setting validation dataset using the same IDataView as training dataset which avoids the train-validation split by AutoML API.

You are right! I used this SQL query

SELECT 
CAST(DecimalColumn1 as real) as [DecimalColumn1], 
CAST(IntColumn2 as real) as [IntColumn2], 
CAST(IntColumn3 as real) as [IntColumn3], 
CAST(IntColumn4 as real) as [IntColumn4], 
[LongStringColumn1], 
[Name] as [Name], 
[ShortStringColumn3] as [ShortStringColumn3], 
[LabelColumn] as [LabelColumn] 
FROM IAi

UNION ALL
SELECT
CAST(NULL as real) as [DecimalColumn1], 
CAST(8 as real) as [IntColumn2], 
CAST(0 as real) as [IntColumn3], 
CAST(NULL as real) as [IntColumn4], 
'Kraków' as [LongStringColumn1], 
'Delegacja krajowa | Alicja Konieczny' as [Name], 
NULL as [ShortStringColumn3], 
'Szkolenie' as [LabelColumn]

UNION ALL
SELECT
CAST(NULL as real) as [DecimalColumn1], 
CAST(8 as real) as [IntColumn2], 
CAST(0 as real) as [IntColumn3], 
CAST(NULL as real) as [IntColumn4], 
'Kraków' as [LongStringColumn1], 
'Delegacja krajowa | Alicja Konieczny' as [Name], 
NULL as [ShortStringColumn3], 
'Szkolenie' as [LabelColumn]

UNION ALL
SELECT
CAST(NULL as real) as [DecimalColumn1], 
CAST(8 as real) as [IntColumn2], 
CAST(0 as real) as [IntColumn3], 
CAST(NULL as real) as [IntColumn4], 
'Kraków' as [LongStringColumn1], 
'Delegacja krajowa | Alicja Konieczny' as [Name], 
NULL as [ShortStringColumn3], 
'Szkolenie' as [LabelColumn]

UNION ALL
SELECT
CAST(NULL as real) as [DecimalColumn1], 
CAST(73 as real) as [IntColumn2], 
CAST(0 as real) as [IntColumn3], 
CAST(NULL as real) as [IntColumn4], 
'Berlin' as [LongStringColumn1], 
'Delegacja zagraniczna | Administrator Systemu' as [Name], 
NULL as [ShortStringColumn3], 
'Konferencja' as [LabelColumn]

And training was done without any doubts.
Maybe there's option to add "human readable" error message in upcoming version?

[rosebyte]

Closed as answered.