Add code coverage to Janssen

[ossdhaval]

Currently code coverage is being reported only on few Janssen repositories via Codecov or Codacy.

Need to add code coverage for repositories where it is applicable.

Benefit: This is an important quality metric which tells us how robust our automated unit and integration testing frameworks are.

---

Due: 31st May 2021

[ossdhaval]

Successfully integrated code coverage for fork of jans-auth-server with codacy. Now Codacy is reporting code coverage.

[ossdhaval]

After integrating code coverage, when I looked at Codacy code analysis and coverage reports I found them lacking in few key aspects. Few of them are as below:

• Doesn't present code in a hierarchical structure. Just lists file names with relative path. So, if you want to see which package is having least code coverage, you can't.
• On the coverage detail page, Codacy doesn't give clear idea about how many lines of code that class has. This is important as smaller classes tend to have more coverage and larger classes tend to lag. Aspect of size is missing in the file listing.
• Doesn't show branch/condition coverage anywhere

So, I am evaluating better alternatives.

After Codacy, second tool that we are familiar with is Codeov.com. But this is only code coverage tool and not static code analysis. What we need is a tool that gives a complete view of code quality, all at one place. So, eliminating this option.

Third tool is SonarCloud. Below is comparative feature analysis of Codacy and SonarCloud.

feature	Codacy	SonarCloud
Configurability of rules to analyse code	Y	Y
Github badge	Y	Y
Security issues reporting	Y	Y
Github integration : run on PR creation, commit, merge etc	Y	Y
Integration with Jenkins	Y	Y
Issue and coverage trend reports	Y	Y
Bug reporting inside Github PR	Y	Y
Individual PR analysis	Y	Y
Multiple programming language support	Y	Y
Thresholds for issues, complexity, coverage, duplication at commit, PR, repo level	Y	N
Number of lines on listing page	N	Y
Branch coverage reporting	N	Y
Collaborative features like reviewing on PR from platform	N	Y
Show consolidated view of repository by merging issues and coverage of all underlying modules	N	Y

Few more points:

• IDE integration: Sonar has better IDE integration via SonarLint plug-in. Specially because it can connect to central SonarCloud project from where we can enable/disable rules which would then percolate to IDEs of all contributing developers. This is not possible in Codacy where you have to use PMD plug-in to see analysis locally in IDE.
• Lower learning curve: Sonar and SonarLint are widely used and most of the developers are
• Rule set: SonarCloud seems to have more robust rule set. When I tried both tools on few obvious issues (like possible NullPointer), Sonar did catch it but not Codacy.
• Inline issue reporting in PR:
  Codacy is better at this. It annotates issues in code within Github itself. So, you don't have to leave your PR page on Github to check what issues were reported. Against this, Sonar provides summary of issues found in PR conversion tab with links that take you to SonarCloud. Once you are on SonarCloud, you can see code annotated with issues and give comments. This feature is being worked by SonarCloud team and would be available in some future release.

My view is that we should use SonarCloud instead of Codacy for Janssen. Please do chime in with your thoughts.

[ossdhaval]

Post monorepo move:
Plan is to move away from Jenkins CI-CD and use GH actions. We need to be able to run tests and report code coverage to sonar for further reporting.