CANN/ge ST测试用例开发指南

ST Case Development Guide

【免费下载链接】ge GE（Graph Engine）是面向昇腾的图编译器和执行器，提供了计算图优化、多流并行、内存复用和模型下沉等技术手段，加速模型执行效率，减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力，并同时支持 onnx、pb 等主流模型格式的解析与编译。 项目地址: https://gitcode.com/cann/ge

Return to DT Test Case Development Guide

ST's testing scope is a feature, enabling and testing a feature from a large module perspective (such as compile-time, runtime), and observing whether compilation/execution results meet expectations. ST cases correspond to STC, so compared to UT cases, ST cases are more formal and correspondingly fewer in number.

ST Case Annotation Requirements

Each ST needs to correspond to an STC design. ST annotations are the STC's description of the case. During requirement design, STC is carried via excel and other methods. After coding is complete, STC design is carried through ST case annotations. Subsequent changes and additions to ST cases should synchronously modify corresponding annotations. STC original excel files are no longer maintained. ST case annotations need to include four parts:

• Case description: One sentence describing this case's test scenario
• Preconditions: How to construct the ST environment, which basic environment data to fake and stub
• Test steps: Brief description of test steps
• Expected results: What results are expected. If there are multiple items, write them separately

For example, if you want to test whether the If node executes normally in rt2, you might have the following ST case annotation:

/**
 * Case description: Test that when the cond input of the If operator is a scalar with value 1, execution enters the then branch
 * Preconditions:
 *   1. Fake Shape and Rank operator's lowering and execution kernel
 * Test steps:
 *   1. Construct an If graph, the then branch contains a Shape node, the else branch contains a Rank node
 *   2. Lower and load this If graph
 *   3. Attach ESS to monitor the execution graph's execution status
 *   4. Construct two Data, one is scalar value 1 passed as cond; another is Tensor with shape {1,1}
 *   5. Execute
 * Expected results:
 *   1. Network output result is {2}
 *   2. Through ESS observation, Rank was not executed, Shape was executed
 */

ST Case Scenario Design Checklist

How to Validate

Graph Compilation Validation

The graph compilation process is divided into graph preparation, graph optimization, and graph compilation from a process perspective. The first two stages mainly modify the graph, and the last stage completes operations such as stream allocation, task generation, and memory allocation (for static models). Certain key attributes are also saved on the graph.

Graph Validation

During the graph optimization process, graphs are dumped stage by stage. You can validate nodes, edge relationships, attributes, etc. on the graph by dumping a certain stage. Use DUMP_GRAPH_WHEN and CHECK_GRAPH macros together. The first macro specifies which stage's graph to dump, and the second macro can write graph validation logic.

/**
 * 
 *       Variable(2, 3, 4, 5)                      Relu3(1,2,3,4,5)
 *          /            \                             /        \
 *     TransData      TransData                    TransData    summary
 *         |              |                            |
 *   Relu(1,2,3,4,5)  Relu(1,2,3,4,5)            Variable(2, 3, 4, 5)
 *          \            /                             |
 *             NetOutput -------------------------------
 * 
 * Case description: After Summary optimization, Summary operator is deleted, its input node connects to Netoutput as network output
 * Preconditions:
 * 1. Register Summary operator's graph
 * Test steps:
 * 1. Construct a graph with Summary operator
 * 2. Specify DUMP PreRunAfterHandleSummaryOp this graph
 * 3. Construct Session, execute AddGraph and RunGraph interfaces, execute graph compilation operation
 * 4. Validate against PreRunAfterHandleSummaryOp graph
 * Expected results:
 * 1. On PreRunAfterHandleSummaryOp graph, Summary operator is deleted
 * 2. Netoutput's outputs increase from 2 to 3. Relu3 connects to Netoutput as output
 * 3. SummaryCallback function was called once
*/
TEST_F(SummaryAndCheckPointTest, test_optimize_summary_graph) {
  // build graph
  Graph graph = BuildVariableGraph();

  // new session & add graph
  map<AscendString, AscendString> options;
  options[ge::OPTION_GRAPH_RUN_MODE] = AscendString("1"); // train flag
  options[VARIABLE_MEMORY_MAX_SIZE] = "12800";
  Session session(options);
  uint32_t graph_id = 1;
  auto ret = session.AddGraph(graph_id, graph, options);
  EXPECT_EQ(ret, SUCCESS);

  // register Summary call back func
  static uint32_t called_count = 0;
  ge::session::pCallBackFunc callbackFuncSummary = [](https://gitcode.com/cann/ge/blob/43f6ad3f1a1a70b7d33f8aa3c5f96f56182072aa/docs/en/contributions/dev_test_guide/uint32_t graph_id, const std::map<AscendString, ge::Tensor> &params_list?utm_source=gitcode_repo_files) {
    called_count++;
    return SUCCESS;
  };
  ret = session.RegisterCallBackFunc("Summary", callbackFuncSummary);
  EXPECT_EQ(ret, SUCCESS);

  // build input tensor
  std::vector<Tensor> inputs;
  Shape shape({1, 1, 224, 224});
  TensorDesc tensor_desc(shape, FORMAT_NCHW, DT_FLOAT);
  std::vector<uint8_t> data(224 * 224 * sizeof(float));
  Tensor tensor(tensor_desc, data);
  inputs.emplace_back(tensor);

  std::vector<Tensor> outputs;
  // build_graph through session
  DUMP_GRAPH_WHEN("PreRunAfterHandleSummaryOp");
  ret = session.RunGraph(graph_id, inputs, outputs);
  EXPECT_EQ(ret, SUCCESS);
  // check result
  CHECK_GRAPH(PreRunAfterHandleSummaryOp) {
    auto summary = graph->FindNode("summary");
    ASSERT_EQ(summary, nullptr); // summary has been deleted
    auto netoutput = graph->FindNode("output");
    ASSERT_NE(netoutput, nullptr);
    ASSERT_EQ(netoutput->GetInDataNodes().size(), 4); // add relu3 as input of netoutput
    ASSERT_EQ(netoutput->GetInDataNodes().at(3)->GetName(), "relu3");
  };
  EXPECT_EQ(called_count, 1);
}

DUMP_GRAPH_WHEN Available Phase List

The following are DUMP_GRAPH_WHEN phase names actually used in existing cases. When writing ST cases, you can select appropriate phases to capture graph snapshots as needed:

This list is derived from actual usage in existing cases and may be incomplete. If you need to dump new phases, you can register corresponding dump phase names in production code.

Graph Execution Validation

For graph execution, you need to construct a corresponding network and produce different results based on different inputs. ST cases directly validate the network's output results.

void RunIfGraph2(TensorHolder &pred_tensor, bool expect_branch) {
    // Construct an If graph
    auto compute_graph = ShareGraph::IfGraph2();
    ASSERT_NE(compute_graph, nullptr);
    compute_graph->TopologicalSorting();
    GeModelBuilder builder(compute_graph);
    auto ge_root_model = builder.BuildGeRootModel();

    // Stub Shape/Rank operators, take over execution of these two operators
    GertRuntimeStub rtstub;
    rtstub.StubByNodeTypes({"Shape", "Rank"});

    // lowering
    auto exe_graph = ModelConverter().ConvertGeModelToExecuteGraph(ge_root_model);
    ASSERT_NE(exe_graph, nullptr);
    auto model_executor = ModelV2Executor::Create(exe_graph, ge_root_model);
    ASSERT_NE(model_executor, nullptr);

    // Create an observer to observe the execution graph's execution status
    auto ess = StartExecutorStatistician(model_executor);

    // Load
    EXPECT_EQ(model_executor->Load(), ge::GRAPH_SUCCESS);

    // Construct network input stream and tensor
    auto input_holder = TensorFaker().Placement(kOnDeviceHbm).DataType(ge::DT_FLOAT).Shape({8, 3, 224, 224}).Build();
    std::vector<Tensor *> inputs{pred_tensor.GetTensor(), input_holder.GetTensor()};
    auto output_holder = TensorFaker().Placement(kOnHost).DataType(ge::DT_INT64).Shape({8}).Build();
    std::vector<Tensor *> outputs{output_holder.GetTensor()};
    rtStream_t stream;
    ASSERT_EQ(rtStreamCreate(&stream, static_cast<int32_t>(RT_STREAM_PRIORITY_DEFAULT)), RT_ERROR_NONE);
    auto stream_value = FakeValue<uint64_t>(reinterpret_cast<uint64_t>(stream));

    // Expected network output value, because this is a general test function,
    // the expected true/false branch determines whether to execute shape or rank operator,
    // thus having different expected output values
    auto shape_count = expect_branch ? 1 : 0;
    auto rank_count = expect_branch ? 0 : 1;
    auto output_shape = expect_branch ? Shape({4}) : Shape({});
    auto output_data = expect_branch ? Shape({8, 3, 224, 224}) : Shape({4});

    // Execute
    ess->Clear();
    ASSERT_EQ(
        model_executor->Execute({stream_value.value}, inputs.data(), inputs.size(), outputs.data(), outputs.size()),
        ge::GRAPH_SUCCESS);
    // Validate Shape/Rank node execution count meets expectations
    EXPECT_EQ(ess->GetExecuteCountByNodeTypeAndKernelType("Shape", "FakeExecuteNode"), shape_count);
    EXPECT_EQ(ess->GetExecuteCountByNodeTypeAndKernelType("Rank", "FakeExecuteNode"), rank_count);
    EXPECT_EQ(output_holder.GetTensor()->GetShape().GetStorageShape(), output_shape);
    EXPECT_EQ(output_holder.GetTensor()->GetShape().GetOriginShape(), output_shape);
    EXPECT_EQ(memcmp(output_holder.GetTensor()->GetAddr(), &output_data[0], output_data.GetDimNum() * 8), 0);

    // Cleanup resources
    ASSERT_EQ(model_executor->UnLoad(), ge::GRAPH_SUCCESS);
    rtStreamDestroy(stream);
  }

Two-Round Execution Validation

Sometimes a business distinguishes between initialization and running states, for example:

• Graph execution distinguishes between graph loading and graph execution phases. First execution does loading and execution, subsequent executions only do execution
• For end-to-end execution of a computational graph, first execution goes through compilation, loading, and execution phases. Second execution skips compilation and loading, only doing execution

For such businesses, consider making two rounds of calls to the running state and validating the results of both rounds separately.

The reason for doing two rounds of validation is that for such businesses, during the first round of execution, the system is in an initialized state. After the first round, there may be dirty data in the system that affects the second round's execution results. Making two rounds of calls can effectively validate dirty data issues.

  // First execution
  ASSERT_EQ(model_executor->Execute({i3.value}, inputs.GetTensorList(), inputs.size(),
                                    outputs.data(), outputs.size()),
            ge::GRAPH_SUCCESS);
  // More validation details omitted here

  // Second execution
  ASSERT_EQ(model_executor->Execute({i3.value}, inputs.GetTensorList(), inputs.size(),
                                    outputs.data(), outputs.size()),
            ge::GRAPH_SUCCESS);
  // Same detailed validation as after first execution

  // Model unload and destroy
  ASSERT_EQ(model_executor->UnLoad(), ge::GRAPH_SUCCESS);
  rtStreamDestroy(stream);

【免费下载链接】ge GE（Graph Engine）是面向昇腾的图编译器和执行器，提供了计算图优化、多流并行、内存复用和模型下沉等技术手段，加速模型执行效率，减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力，并同时支持 onnx、pb 等主流模型格式的解析与编译。 项目地址: https://gitcode.com/cann/ge