erlang CodeCoverage Elixir的更多指标

s5a0g9ez  于 2022-12-08  发布在  Erlang
关注(0)|答案(2)|浏览(172)

Background

I have a test suite and I need to know the coverage of the project. I have played around with mix test --cover but I find the native erlang's coverage analysis tool to be insufficient at best.
The native coverage tool doesn't tell you about branch coverage nor function coverage. It's only metric seems to be relevant lines which I have no idea how they calculate. For all I know, this is just the most basic form of test coverage: see if a given text line was executed.

What have you tried?

I have tried Coverex but the result was disastrous. Not only does it suffer from the same issues that the native tool does, it also seems not produce correct results as it counts imported modules as untested.
Or maybe it is doing a great job and my code is poorly tested, but I can't know for sure because it doesn't tell me how it is evaluating my code. Have 40% coverage in a file? What am I missing? I can't know, the tool wont tell me.
I am now using ExCoveralls . It is considerably better than the previous options, it allows me to easily configure which folders I want to ignore, but it uses the native coverage tool, so it suffers pretty much from the same issues.

What do you want?

I was hoping to find something among the lines of Istanbul, or in this case nyc :
https://github.com/istanbuljs/nyc
It's test coverage analysis tells me everything I need to know, metrics and all:

Branches, Functions, Lines, Statements, everything you need to know is there.

Questions

  1. Is there any tool that uses Istanbul for code coverage metrics with Elixir instead of the native erlang one?
  2. If not, is there a way to configure the native coverage tool to give me more information?
  3. Which metrics does the native coverage tool uses ?
44u64gxh

44u64gxh1#

原生覆盖工具在源代码的每一行插入“bump”调用,记录模块、函数、arity、子句号和行号:

bump_call(Vars, Line) ->
    A = erl_anno:new(0),
    {call,A,{remote,A,{atom,A,ets},{atom,A,update_counter}},
     [{atom,A,?COVER_TABLE},
      {tuple,A,[{atom,A,?BUMP_REC_NAME},
                {atom,A,Vars#vars.module},
                {atom,A,Vars#vars.function},
                {integer,A,Vars#vars.arity},
                {integer,A,Vars#vars.clause},
                {integer,A,Line}]},
      {integer,A,1}]}.

(from覆盖
上述函数插入的代码为:

ets:update_counter(?COVER_TABLE,
  {?BUMP_REC_NAME, Module, Function, Arity, Clause, Line}, 1)

也就是说,将所讨论的给定模块/函数/行的条目递增1。在所有测试完成后,cover将使用此表中的数据,并显示给定行的执行次数。
正如the cover documentation中所提到的,你可以获得模块、函数、函数子句和行的覆盖率。看起来ExCoveralls只在它的报告中使用行覆盖率,但没有理由它不能完成所有四种类型的覆盖率。
不支持分支覆盖范围。似乎支持分支覆盖范围需要扩展“bump”记录并更新cover.erl以记录该信息。在有人执行此操作之前,仅当分支出现在不同行上时,覆盖范围信息才是准确的。例如:

case always_false() of
    true ->
        %% this line shows up as not covered
        do_something();
    false ->
        ok
end.

%% this line shows up as covered, even though do_something is never called
always_false() andalso do_something()
ct2axkht

ct2axkht2#

To add to @legoscia excellent response, I also want to clarify why cover does not do statements evaluation. According to this discussion in the official forum:
https://elixirforum.com/t/code-coverage-tools-for-elixir/18102/10
The code is first compiled into erlang and then from erlang into a modified binary file (but no .beam file is created) that is automatically loaded into memory and executed.
Because of the way erlang code works, a single statement can have several instructions:
and single line can result in multiple VM “statements”, for example:

Integer.to_string(a + 1)

Will result with 2 instructions:

{line,[{location,"lib/tasks.ex",6}]}.
{gc_bif,'+',{f,0},1,[{x,0},{integer,1}],{x,0}}.
{line,[{location,"lib/tasks.ex",6}]}.
{call_ext_only,1,{extfunc,erlang,integer_to_binary,1}}.

Therefore it is rather tricky for an automatic analysis tool to provide statement coverage because it is hard to match statements to instructions, especially as in theory a compiler is free to reorder commands as it pleases as long as the result is the same.

相关问题