我通过编写一个自定义gradle任务解决了这个问题。
以下是我的解决方案：
Gradle任务：

task aggregatReport(type: JavaExec) {
    println("Running aggregatReport task")
    description 'Combine multiple cucumber report json file into one json file'
    classpath sourceSets.main.runtimeClasspath
    mainClass = "com.ReportAggregator"
    def reportFolderPath = "./build/cucumber-reports/";
    File file = file(reportFolderPath);
    args file.toPath().toAbsolutePath().toString()
}

代码：

public class ReportAggregator {
    public static void main(String[] args) throws IOException {
        System.out.println("ReportAggregator with argument -> " + Arrays.toString(args));
        ObjectMapper objectMapper = new ObjectMapper();
        File reportFolder = new File(args[0]);
        System.out.println("Report folder  -> " + reportFolder.isDirectory() + " " + reportFolder.exists());
        List<Path> reportFiles = Files.walk(reportFolder.toPath()).filter(file -> file.toFile().getName().endsWith(".json")).toList();
        List<Map<String, Object>> finalReportList = new ArrayList<>();
        if (CollectionUtil.isNotEmpty(reportFiles)) {
            System.out.println("Total report file ->" + reportFiles.size());
            reportFiles.forEach(path -> {
                try {
                    String jsonString = FileUtils.readFileToString(path.toFile(), Charset.defaultCharset());
                    TypeReference<List<Map<String, Object>>> typeRef = new TypeReference<>() {
                    };
                    List<Map<String, Object>> hashMaps = objectMapper.readValue(jsonString, typeRef);
                    finalReportList.addAll(hashMaps);
                } catch (IOException e) {
                    System.err.println("Can not read file -> " + path.toFile().getAbsolutePath());
                }
            });

            if (CollectionUtil.isNotEmpty(finalReportList)) {
                String finalReportFileName = reportFolder.getAbsolutePath() + "/final.json";
                File finalReportFile = new File(finalReportFileName);
                if (finalReportFile.exists()) {
                    boolean delete = finalReportFile.delete();
                    System.out.println("File is deleted -> " + delete);
                }
                FileUtils.writeStringToFile(finalReportFile, objectMapper.writeValueAsString(finalReportList), Charset.defaultCharset());
                System.out.println("Report file created");
            } else {
                System.err.println("Empty report is created");
                throw new ReportException("Empty report created");
            }
        } else {
            throw new ReportException("No report file found under folder ->" + reportFolder.getAbsolutePath());
        }
    }
}

和gralde任务相关性设置：

test {
    println("Running test task")
    useTestNG() {
        suites 'src/test/testng.xml'
    }
    finalizedBy aggregatReport
}

task importJunitResultsToXrayCloud() {
    dependsOn 'test'
    description 'Imports Cucumber test results to Xray Test Management for Jira Cloud.'

    doLast {
       //code to upload report
    }
}

上传报告只需运行./gradlew importJunitResultsToXrayCloud