SymTex-ASC

评测集介绍

Introduction

SymTex-ASC是一个用于评估大语言模型在符号非单调推理领域能力的数据集，包含540个样本。ASC（Answer Set Computation）任务是一个生成任务，给定非单调推理程序（包含事实、规则），要求输出所有非事实可推到结论。

Meta Data

facts: 事实集合

rules: 规则集合

answer set: 非事实结果集合

[输入]

facts:

      Angela_Owen, James_Burgess and Maria_Cline are sure.

      Angela_Owen, James_Burgess and James_Burgess are not sure.

      Angela_Owen, Maria_Cline and Maria_Cline are sure.

      Maria_Cline and James_Burgess are street.

      Maria_Cline and Angela_Owen are street.

      James_Burgess, Angela_Owen and Angela_Owen are sure.

      Maria_Cline, Maria_Cline and Angela_Owen are sure.

 

rules:

      [If] A and A are street; B, A and C are not sure, [then] B, C and A are can

      [If] B, A and C are sure, [then] B, A and C are wait

      [If] B, A and C are sure; C and B are computer, [then] C and B are not compare

      [If] B, A and C are wait, [unless] B, C and A are can, [then] A, C and B are not as

      [If] B, A and C are wait; B, A and C are not sure, [unless] C and B are compare, [then] C and B are information

      [If] C and B are information; B, A and C are not sure, [then] C and B are strategy

      [If] C and B are strategy; C and B are compare, [then] C, B and C are wait

 

[预期输出]

answer set:

      Angela_Owen, James_Burgess and Maria_Cline are wait

      Angela_Owen, Maria_Cline and Maria_Cline are wait

      James_Burgess, Angela_Owen and Angela_Owen are wait

      Maria_Cline, Maria_Cline and Angela_Owen are wait