Finding superfluous C/C++ #includes

Unnecessary #include statements can increase build times, make code harder to understand, and make restructuring more difficult. It is trivial to find these with Structure101 Studio, so you can remove them from the code, or from the model prior to further analysis and restructuring.

Finding redundant #includes means finding file-to-file dependencies that contain an #include, but whose weight is 1 – indicating that #include is the only dependency between the files.

Start by loading your project into Studio using the “Detail” granularity and “Compiler” model options:

You’re going to show the file slice on the Structure tab, so it helps to switch off dependencies for now. To do this, click on this button (not the dropdown next to it) so that it is deselected:

Now select the file slice using the slice button. This causes all the files to be displayed in the LSM viewing area:

Now go to the panel below the LSM and select the Map Contents tab, and the Dependencies sub-tab. This shows all the dependencies on the LSM (not currently shown, but they are there). Click once on the “weight”, then on the “Primary usage” column header – this sorts the table of dependencies first on the primary usage, then on the weight. This means that all #includes of weight 1 are in a contiguous block in the table. Scroll down to find these:

Now you can multi-select all of the redundant #includes by clicking the first, and scrolling down to, and shift-clicking on the last.

If you wish to export the sublist, right-click/copy-all will move them to the clipboard:

… from whence you can paste to a spreadsheet, or Jira, or wherever you wish…

And/or if you wish to continue analysing and restructuring activities in advance of the source code catching up, you can remove them from your Structure101 project by clicking on the “cut” button while the dependencies are still selected:

When you remove the dependencies in the Structure101 project, all the remove actions are added to the current Action list:

These are re-applied whenever you open or update the project with this action list selected. This means that you can now switch to a lower granularity model (for example if it’s a huge project, this can improve UI responsiveness). Go to the project settings and change the granularity:

After changing the granularity, the project will be loaded with the lighter model (no file internals), but the superfluous #includes will be re-removed by the action list. Note that you cannot remove redundant #includes in overview (or includes-only) granularity because you need to see the function-level dependencies in order to tell which file-level are redundant.

By the way, while you are tidying up the #includes, you may want to look at all the remaining dependencies for which there is no corresponding direct #include. Still in the file slice, and with the “Map contents”/”Dependencies” displayed, any dependencies whose “primary usage” is not “includes” are inter-file usage for which there is no #include statement…

Some of the features shown above are relatively recent to Structure101 Studio, so if you’re not seeing something used, you’ll need to grab the latest release.

Polyglot Architecture Control and its Gotchas

This guest post was originally published on LinkedIn and is included here with kind permission.

 

The rise of polyglot programming

 

The need for software on all sorts of domains (“software is eating the world”) has pushed the need for more developers, fast. As more professionals are hired, better scrutiny is necessary if we are to build malleable software which can evolve and constantly adapt – see “Building Evolutionary Architectures”.

With the current trend of single-page web applications (usually written in Javascript- or Typescript-based frameworks and libraries such as React, AngularJS among others), the concern for software maintenance on the front-end has grown. How can we define a layered and componentized front-end structure and make sure that only code which conforms to this structure is accepted? What kinds of Fitness Functions can be used to perform architecture control on the front-end?

On the back-end the pain is no different. Companies usually have legacy monoliths which they have been trying to strangle into smaller services (or at least into a monolith with a plugin-based architecture). Very often the programming language used for services is different from the original monolith, and the array of languages for the back-end increases inside the company. Python, Java, C#, TypeScript, Go are some of the popular ones. The quest for architectural fitness functions still applies. Can it be done in a uniform way?

A Quick Definition of Architecture

If you still haven’t watched “Real Architecture: Engineering? or Pompous Bullshit?” you should (slides here). It offers a more precise definition and scope of the term “architecture” in the software domain, with some pearls of wisdom such as:

“The architecture is there to satisfy requirements. Architecture that never refers to necessary qualities, performance characteristics, costs, and constraints Is not really architecture. Of any kind.”

 

Here we will focus on a narrower concept of architecture, as seen in materials from Bob Martin and others (see this blog post and this book, for example). More specifically, we want to focus on a subset of the qualities and constraints – we want to focus on features that facilitate maintenance, sustainability and evolution.

Addressing the Company Culture

At Softplan we have been using a maturity assessment of teams inspired by martial arts and belts. Areas of expertise (Customer Success, User Experience, DevOps Culture, etc) are assessed according to different levels of difficulty and a colored pin is given to that team. As teams move up in their maturity, they are awarded cash for a nice barbecue party for the team, among other things:

 

Softplan Team

 

One of these areas of expertise which is assessed is architecture skills (aiming at malleability and software maintenance), which we divide into two main kinds:

1) “Micro Architecture”

Here we look into things like Cyclomatic Complexity, number of lines of code in a method/function and other metrics which reflect the tidiness of the code when looked at “up close”. We use a little tool we wrote called srccheck to generate histograms and interactive svg scatter plots which can be used by tech leads to keep an eye on these values.

 

CyclomaticStrict

MaxNesting

 

Unlike other tools which only allow a limit on the maximum amount of a metric to fail a build or merge request, we can perform filtering on max, median, standard deviation, average, variance, etc. This allows teams to tolerate exceptional cases of a higher metric value while constraining the remaining elements into smaller values via smaller median and variance. This is particularly useful when strangling legacy or for optimized code with outliers.

Currently srccheck supports most languages which SciTools Understand supports: Java, C#, Delphi, Python, JavaScript/Typescript (“Web” in their tooling), C/C++.

2) “Macro Architecture”

Here we look at how well organized the layers and components are. We look at two main variables – Tangle and Fat, both supported by Structure101 and nicely plotted by their tool:

Structure101

 

For micro and macro architecture, teams commit to a given maturity level by incorporating metric values for Fat, Tangle, Cyclomatic Complexity etc in their GitLab pipelines. Any merge request is fed through the pipeline, then these fitness functions are executed (Structure101 Build is invoked from the command-line) and code is rejected if it is below the agreed upon maturity level. Here is an example of such a pipeline:

 

Kaloi

 

Note how we have Kaloi steps for both micro and macro architecture – after the UDB creation phase.

In order to support microarchitecture control under polyglot programming we wrote a Structure101g “plugin” (flavor) named net.betterdeveloper.understand which takes the SciTools Understand .udb file and outputs an XML for Structure101g (behind the scenes, transparent to the user). Structure101g then does its work and enables architecture control for Python, JavaScript, Typescript, Delphi and C++ just like Structure101‘s native versions for Java and C#.

Here is an example – separating reducers from actions in a JavaScript project:

 

JavaScript1

Here we can see the dependencies between reducers and actions:

JavaScript2

Lessons Learned

In this process of macro architecture control over the years we have learned a few lessons, some of them specific to each programming language. We’ll share a few.

Python

The community has many different levels of tutorials, some of which induce people into writing tangled code. One such example is the use of an SQLAlchemy plugin for Flask, which will introduce a tangle in your code, making Models depend on Controllers (REST layer) and vice-versa. We have presented this problem at Python Brasil 2016 (the slides in Portuguese are here and the screencast is here). Borrowing some images from that presentation, here is the DSM notation of the dependencies when the Flask-SQLAlchemy plugin is used:

 

DSM

 

The matrix should be sub-diagonal and it isn’t – indicating a tangle in the code (between controller and domain). Fortunately, a GitLab pipeline was used and the issue was detected:

 

GitLab Pipeline

 

After fixing the tangle, the correct DSM looks like this (no tangles):

 

DSM No Tangles

 

And the pipeline is green:

 

Build Succeeded

 

 

Green Pipeline

 

Another issue we see in Python projects is the abuse of __init__.py in a package. It can refer to elements in subpackages, but these subpackages have an implicit import dependency on __init__.py of all parent packages, leading to a circular dependency (tangle) detected by Understand and exposed in Structure101g via our net.betterdeveloper.understand flavor/plugin. This cultural habit was a problem to deal with. For greenfield projects it was not an issue – it can be prevented with Structure101g, rejecting such code. But for Python legacy code, it meant running Structure101g Build in “reject code which increases the number of tangles, but accept existing ones” mode until we could pragmatically get rid of the tangles. You really want to enable Structure101g in your project at day 1 to avoid this kind of pain. Here is an example of tangle detected when the team took too long to instrument the build pipeline with Structure101g at Softplan:

 

Tangled1

Tangled2

 

Tangled3

 

But this sort of problem is not exclusive to Softplan. Here is Django itself:

 

Django

 

You can see __init__.py tangles, among others.

The lesson here is that as Python becomes more widespread in polyglot setups, architecture control with Structure101g becomes a must-have.

Java and C#

Structure101 Studio for Java and Structure101 Studio for .NET take compiled binaries as input. However, some tools such as Lombok perform some code generation and/or weaving which introduce tangles in the generated code. Fortunately, Structure101 has excellent configurability and these can be excluded. Just keep an eye for them.

Another point that deserves attention is the fact that some Java projects prefer to use com.mycompany.entity.layer (e.g. com.softplan.book.model, com.softplan.book.view, etc) instead of com.mycompany.layer.entity (e.g. com.softplan.model.book, com.softplan.view.book, etc) and this can produce tangles in automatic mode, when you assume that packages represent layering. Fortunately, you can use Model Transformations among other Structure101 features to better model the logical layering of your way of modeling things in code. You may prefer to align your components with the POM files instead of packages – all supported by the tool, fortunately.

Another issue for us was the fact that some Java projects put pure interfaces in a “parent package” and Impl classes in a sub-package (even though, strictly speaking, the package model in Java is not truly hierarchical – unlike C# or Python which are truly hierarchical). This may produce artificial tangles where clients import the interfaces but the dependency sees an import into the concrete implementation, leading to an undesired structure. Separating APIs and IMPLs into “brother packages” achieves true separation and solves this issue – albeit with resistance from some of our Java developers. Model Transformations is another possibility, which can avoid code changes.

JavaScript and Typescript

Convincing pure front-end developers that architecture matters and there are tools for that can be time consuming. It generally isn’t a problem for developers with previous Java background, though. Agreeing on a structure for folders in the project is also time consuming. Due to the fact that the language does not have a native namespace/package mechanism like Java, C# and Python, the folder structure takes that role (the same is done when analyzing C/C++ code in our net.betterdeveloper.understand Structure101g flavor). However, some front-end developers like to structure projects by feature and others by layers or even components. The real challenge is being able to cover all of these – something that can be done with Model Transformations and also multiple architecture rules in Structure101g. In other words, your problems won’t be technical. The real challenge will be changing the culture and achieving consensus among people on the same team and across teams in the same company. For example, below we show two separate architecture rules from two separate teams using React.

Team A organized things like this:

 

React1

 

Team B organized things like this:

 

React2

 

One desired feature of the “Team B” approach above is the use of a Model Transformation to collect all tests into a logical folder/package named “tests” at the top. This is very useful and important both for JavaScript/Typescript as well as Python, where developers often put test code in the same folders as production code, which can cause tangles between production code and tests. By ensuring there is a logical component/folder for tests and putting it above all else in the architecture ensures that no production code uses test code. An alternative to a Model Transformation is the use of a specific architecture rule/diagram for this constraint (“no production code can use test code”).

Our challenge now is to reach consensus and make sure the macro architecture and layers for the different React projects follow the same rationale, with the best insights from the multiple teams.

On the other hand, the good news is that our mobile products are written with React Native, so this sort of architecture control is just as valuable for mobile development:

 

React Native1

React Native2

React Native3

 

But note that React Native is yet another case where there is much in common with non-mobile, so some architectural rules and sub-diagrams should be the same. More peopleware work is necessary here to reach consensus.

C and C++

Previously at Audaces (circa 2009) we did a similar effort around its Vestuario CAD and IDEA products, among others. Structure101g with our custom flavor allowed us to refactor the tangled CAD monolith into a plugin-based architecture using SOF. The main challenge was in converting a legacy folder structure that was very tangled (all .h files in a single “includes” directory) into a more componentized structure where each component was in a folder and had its own includes and impl subfolders. After all, traditional C/C++ code does not use namespaces like Java – it uses folders to organize code. The component structure becomes all tangled if not properly designed. Structure101g was used to prevent more tangles from being added, while continuous builds with Jenkins allowed the team to converge to the better architecture designed in the tool.

A major gotcha with C and C++ analysis is due to the fact that elements are declared in one file (.h) and implemented in another file (.c, .cpp) so when a dependency exists, one has to be careful to properly detect if it is a dependency into the .h (better, API) or into the .c file (worse, implementation detail). Mastering this is key for a proper C/C++ redesign into a better architecture with regards to tangle and fat.

Here is an example of Structure101g with our net.betterdeveloper.understand flavor analyzing an Open Source flight simulator in C++ and showing tangles when folders are considered the modules of the system:

 

C++

 

At Softplan we don’t have C++ based products or microservices, though.

Kotlin, Go, Ruby, PHP

The lack of support for Kotlin in both Structure101 and Understand is a limiting factor for Kotlin adoption for us. Analyzing JVM bytecodes can be misleading, much like with Lombok for Java, but at a much higher rate. This is unfortunate, as many of our Java developers see new C# features which they wish they could use on the JVM, and Kotlin would be an easy route for that.

Go is not supported by Understand, but has an experimental Structure101g plugin which seems to be in suspended mode. We haven’t put any effort ourselves into writing a custom Go plugin due to lack of time, unfortunately. However, we do have Go initiatives at Softplan and we are deciding what to do. Ideally, we would like to see Understand support Go as well.

Ruby is not supported by Understand, so the same applies. Fortunately, we have no Ruby projects at Softplan, so we don’t need to deal with this problem.

PHP, on the other hand, is supported by Understand. In theory our net.betterdeveloper.understand flavor could be extended to analyze PHP information as well. However, we have put no effort into this, as we have no PHP back-ends at Softplan.

Delphi

Softplan does have legacy application servers and desktop software written in Delphi (Pascal), which are being decommissioned and replaced by smaller back-end services (REST and GraphQL) using the languages mentioned above, interacting with React-based web fronts. Using Structure101g over the years allowed a better understanding and control of the components to prevent further decay when new team members made mistakes introducing new component circularities – Jenkins builds allowed such anomalies to be detected as soon as they happened.

Like C and C++, traditional Delphi code does not make use of packages/namespaces, relying instead on folders to organize the code. By using a monorepo approach the company unfortunately facilitated the abuse of imports and tangles proliferated before Structure101g was brought into the scene.

The fact that folders have a shallow structure in traditional Delphi projects results in many .pas files in them, which also increases Fat significantly. The traditional approach is to use file prefixes (uaipFoo.pas, uedtBar.pas) so we are evaluating switching to Model Transformations based on these prefixes to better organize the files/units into logical nested packages. Currently XS is high because of tangles and because of these folders with too many .pas files (flat).

Future Work

We will be comparing a layered-based approach versus feature-based approach and component-based approach for front-ends (for a detailed discussion of the topic, see chapter 34 of this book). Hopefully we will turn that into a new article in the near future.

On the Python front, we have implemented a flavor which does not need SciTools Understand to perform its analysis (with the drawback of only capturing dependencies at the file import level). It is quite useful and has been used by close friends with success. If you are interested, drop us a message.

 

About the Author

Marcio Marchini is Chief Architect at Softplan, where he oversees the architectural decisions across products and leads the Enterprise Architecture team, responsible for designing the fitness functions for architecture governance. He has also been working as a consultant over the years helping companies tame their legacy code and prevent their greenfield microservices from becoming mini monoliths quickly. You can reach out to Marcio on LinkedIn .

 

Using Structure101 Build with SonarQube

When using Structure101 Build with SonarQube the detection of new structural issues can be delegated to SonarQube. This simplifies the configuration of Structure101 Build but requires an appropriate SonarQube Quality Gate to be defined. [Build 14507 and later of the Structure101 SonarQube plugin]

Overview

The SonarQube plugin defines seven new rules of type code smell: –
  • Packages should not be Fat
  • Classes should not be Fat
  • Methods should not be Fat
  • Packages should not contain tangled sub-packages
  • Dependencies should comply with the Structure Spec
  • Items should not exist in a specified scope unless they are included in the Structure Spec
  • Dependencies should comply with all enforced Architecture Diagrams
If any of the above are found a SonarQube Issue is created. These issues will be reported as ‘new’ according to your project’s Leak Period configuration.
The plugin uses the output file from the Structure101 Build report-key-measures operation. So this must be configured to run before the SonarQube analysis.

Installing and Configuring the SonarQube Plugin

The Structure101 SonarQube plugin jar file should be copied into the extensions/plugins folder of the SonarQube installation. Any previous version of the Structure101 plugin should be removed and the SonarQube instance restarted.

Create a new Quality Profile with language Java

Change the parent to your existing Java quality profile

Click Activate More to add the Structure101 rules

Filter the rules using Java language and Structure101 repository

Activate each of the rules setting the desired severity

Configuring the Quality Gate

On the SonarQube Quality Gates tab click Create and enter a suitable name in the dialog

Click Add Condition and select On Overall Code then select the relevant metric from the drop down list.

Set the value and click the Add Condition button.

Configuring the Build

It is assumed that SonarQube is already setup for your project

Structure101 Build must run the report-key-measures operation before the SonarQube scanner executes

The SonarQube scanner must have the system property structure101.reportdir set so that the Structure101 SonarQube plugin can find the output file of the report-key-measures operation

NOTES

Your Studio project file containing Structure Spec, Architecture Diagrams and Action Lists should be committed to source control so that it is available to your CI process

When multiple branches are published with the same Structure101 Repository Project name then the report-key-measures operation must be configured to use the local Structure Spec and Architecture Diagrams. (ie those contained in the .hsp file)
This ensures the checks are against the correct spec and diagrams for the branch being built. (The latest repository snapshot cannot be used because it may be for a different branch)

Set these properties – useProjectFileSpec=true and useProjectFileDiagrams=true

(Using the local project diagrams is good practice anyway as it removes the need to manually publish updated diagrams or Action Lists. The hsp file can be updated in Structure101 Studio and committed to source control. The next CI build will then publish the updates to the Repository)

Maven Example

This is an example pom.xml configuration to execute the report-key-measures goal

<plugin>
  <groupId>com.structure101.java</groupId>
  <artifactId>maven-structure101-plugin</artifactId>
  <version>14507</version>
  <configuration>
    <project>maven</project>
    <localProject>structure101-settings.java.hsp</localProject>
    <repository>http://ubuntu-server:8989/s101j/data</repository>
  </configuration>
  <executions>
    <execution>
      <id>s101checkKeyMeasures</id>
      <phase>compile</phase>
      <goals>
        <goal>report-key-measures</goal>
      </goals>
      <configuration>
        <report-key-measures-output-file>target/s101-results/key-measures.xml</report-key-measures-output-file>
        <use-project-file-spec>true</use-project-file-spec>
        <use-project-file-diagrams>true</use-project-file-diagrams>
      </configuration>
    </execution>

The report-key-measures-output-file path must be passed to the SonarQube scanner in the system property structure101.reportdir

For example, mvn sonar:sonar -Dstructure101.reportdir=target/structure101-reports/key-measures.xml

Or  in the Jenkins build step as shown below

If you are using a pipeline script or similar mechanism to call the sonarscanner shell script then the structure101.reportdir property must be passed in using the SONAR_SCANNER_OPTS property. The example below can be used in a Jenkins pipeline script.

env.SONAR_SCANNER_OPTS=”-Dstructure101.reportdir=${WORKSPACE}/target/structure101-reports/key-measures.xml”

Structure101 Build Configuration Example

The build configuration file shown below runs the report-key-measures and publish operations. Note the label argument contains the value “LABEL_TOKEN”. This will need to be replaced with an appropriate value when Structure101 Build is run.

This can be achieved with an execute shell command step

The label is passed to the Structure101 Build command using -Ds101.label. (In this example, a concatenation of the system variables GIT_BRANCH and BUILD_ID separated by an underscore)

java -Ds101.label="${GIT_BRANSH}_$BUILD_ID" -jar /opt/structure101-build-java-all-5.0.14435/structure101-java-build.jar structure101-build-config.xml -licenseDir=/opt/structure101-build-license

(NOTE that any of the arguments can be provided at runtime with -Ds101.<argument name>)


Example build configuration file

<?xml version="1.0" encoding="UTF-8"?>
<headless>
    <operations>

        <!-- report key measures -->
        <operation type="report-key-measures">
            <argument name="output-file" value="const(THIS_FILE)/target/structure101-reports/key-measures.xml"/>
            <argument name="useProjectFileSpec" value="true"/>
            <argument name="useProjectFileDiagrams" value="true"/>            
        </operation>

        <!-- publish snapshot and diagrams -->
        <operation type="publish">
            <argument name="label" value="LABEL_TOKEN"/>
            <argument name="diagrams" value="true"/>
            <argument name="spec" value="true"/>
            <argument name="actions" value="true"/>
        </operation>

    <arguments>
        <argument name="local-project" value="const(THIS_FILE)/structure101-settings.java.hsp">
        </argument>
        <argument name="repository" value="http://ubuntu-server:8989/s101j/data"/>
        <argument name="project" value="Maven"/>
    </arguments>
</headless>

Using Structure101 Build with Jenkins and SonarQube

Structure101 Build and the Structure101 SonarQube plugin can be incorporated into your existing Jenkins CI process. This post describes the necessary configuration. [Build 14507 and later of the Structure101 SonarQube plugin]

Structure101 Build performs architectural checks and can publish a snapshot and the architectural diagrams to a Structure101 repository. The Check Key Measures and Check Spec operations can be configured to fail the build directly, by throwing a run time exception, or indirectly using Jenkins build steps to set the build status.

The SonarQube plugin defines seven new rules of type code smell: –
  • Packages should not be Fat
  • Classes should not be Fat
  • Methods should not be Fat
  • Packages should not contain tangled sub-packages
  • Dependencies should comply with the Structure Spec
  • Items should not exist in a specified scope unless they are included in the Structure Spec
  • Dependencies should comply with all enforced Architecture Diagrams
If any of the above are found a SonarQube Issue is created. These issues will be reported as new according to your project’s Leak Period configuration.
The plugin uses the output file from the Structure101 Build check-key-measures operation. So this must be configured to run before the SonarQube analysis.

 

Installing and Configuring the SonarQube Plugin

The Structure101 SonarQube plugin jar file should be copied into the extensions/plugins folder of the SonarQube installation. Any previous version of the Structure101 plugin should be removed and the SonarQube instance restarted.

Create a new Quality Profile with language Java

 

Change the parent to your existing Java quality profile

Click Activate More to add the Structure101 rules

Filter the rules using Java language and Structure101 repository

Activate each of the rules setting the desired severity

Jenkins Prerequisites

The examples described use several Jenkins plugins that are not installed by default. Check your Jenkins configuration and install any that are missing.

Overview of the Build Steps

The build steps are: –

  1. Execute the compile/test phase
  2. If the build is successful
    1. Update the Structure101 Build configuration file with the label of the current build
    2. Run the Structure101 Build process
      1. Check Key Measures
      2. Check Spec
      3. Publish
      4. Report Summary
  3. Run the SonarQube Scanner
  4. If Check Key Measures or Spec Check failed
    1. Mark the build unstable/failed
  5. If the build is successful
    1. Update the Structure101 Build configuration baseline property
    2. Commit/push the updated Build configuration file

The baseline property in the check-key-measures and check-spec operations is always set to the label of the last build that did not fail the checks. This allows the snapshot to be published on every build, even those that fail the checks. The snapshots capture the model and diagrams for each build and allow visual comparisons between them.

Initial Publish of Snapshot

The Structure101 project must be published to the Structure101 Repository before the Jenkins driven publish operation can run. The label used for this initial snapshot should be added to the structure101-build-config.xml file in the baseline property of the check-key-measures, check-spec and check-architecture operations.

For multiple branches either use different Structure101 project names for each branch or use the same project name and a branch identifier in the label argument of the publish operation.

Build Steps in Detail – Maven

This first example uses the Maven Integration plugin

This plugin adds a Maven Build Step with Pre and Post Steps. No Pre Steps are used in the example. The Post Steps are configured to run only if the build succeeds.

 

The first of the post steps is an execute shell command

This step runs the Structure101 Build process passing in the label using -Ds101.label. (In this example, a concatenation of the system variables GIT_BRANCH and BUILD_ID separated by an underscore)

java -Ds101.label="${GIT_BRANCH}_$BUILD_ID" -jar /opt/structure101-build-java-all-5.0.14435/structure101-java-build.jar structure101-build-config.xml -licenseDir=/opt/structure101-build-license
(NOTE that any of the arguments can be provided at runtime with -Ds101.<argument name>)

 

The second step runs the SonarQube scanner using the Maven goal in an Invoke top-level Maven targets step. Note that the new SonarQube plugin does not require any property definition in the Maven pom.xml. But the plugin does require a system property to be set that provides the path to the output file of the check-key-measures operation run in the previous step. This property is available to the Structure101 plugin when SonarQube invokes its scan.

The path must match that set in the structure101-build-config.xml file

 <operation type="check-key-measures">
<argument name="output-file" value="const(THIS_FILE)/target/structure101/key-measures.xml"/>
<argument name="output-file-for-snapshot" value="const(THIS_FILE)/s101working/structure101report/snapshot-key-measures.xml"/>
<argument name="baseline" value="14"/>
<argument name="useProjectFileSpec" value="true"/>
<argument name="useProjectFileDiagrams" value="true"/>
<argument name="fail-on-architecture-violations" value="false"/>
<argument name="fail-on-fat-package" value="false"/>
<argument name="fail-on-fat-class" value="false"/>
<argument name="fail-on-fat-method" value="false"/>
<argument name="fail-on-feedback-dependencies" value="false"/>
<argument name="fail-on-spec-violation-dependencies" value="false"/>
<argument name="fail-on-total-problem-dependencies" value="false"/>
<argument name="fail-on-spec-item-violations" value="false"/>
<argument name="fail-on-biggest-class-tangle" value="false"/>
<argument name="fail-on-tangled-package" value="false"/>
<argument name="fail-on-architecture-violations" value="false"/>
<argument name="identifier-on-violation" value="S101 key measure violation"/>
</operation>

 

The next step is a Conditional step (single) that uses Text Finder to check the console output for failure messages from the check-key-measures operation and the Set the build result step to set the build status if any are found. In this example the build is set to failed if any of the messages are detected using a simple OR’d regex

The messages are: –

FAIL! fat0 has increased by 
FAIL! fat1 has increased by
FAIL! fat2 has increased by
FAIL! feedbackDependencies has increased by
FAIL! specViolationDependencies has increased by
FAIL! totalProblemDependencies has increased by
FAIL! specItemViolations has increased by
FAIL! biggestClassTangles has increased by
FAIL! tangledDesign has increased by
FAIL! numViolations has increased by

if desired a different build status can be set for the messages by creating multiple Text Finder steps.

 

The next step is also a single conditional that only runs if the build is successful. It executes a shell command that updates the baseline argument of the check-key-measures, check-spec and check-architecture operations in the structure101-build-config.xml file and then commits the changes.

The awk command replaces the line containing the baseline argument with the content of the baseline_tag variable. This should be identical to the variable used to set the label argument.

awk -v baseline_tag="\<argument name\=\"baseline\" value\=\"${GIT_BRANCH}_$BUILD_ID\"/\>" '{gsub("\<argument name=\"baseline\" value\=\".*", baseline_tag)}1' structure101-build-config.xml > baseline-structure101-build-config.xml 
mv -f baseline-structure101-build-config.xml structure101-build-config.xml 
git commit -am "Increment baseline label"

Finally the Git Publisher Post-build Action is used to push the baseline change to the repository.

Build Steps in Detail – Gradle

The gradle example uses a Freestyle project. The steps are all configured as Build Steps.

The Invoke Gradle Script step is followed by a Conditional step (single) that runs only if the build is successful.

This step runs the Structure101 Build process passing in the label using -Ds101.label. (In this example, a concatenation of the system variables GIT_BRANCH and BUILD_ID separated by an underscore)

java -Ds101.label="${GIT_BRANCH}_$BUILD_ID" -jar /opt/structure101-build-java-all-5.0.14435/structure101-java-build.jar structure101-build-config.xml -licenseDir=/opt/structure101-build-license
(NOTE that any of the arguments can be provided at runtime with -Ds101.<argument name>)

 

The next step runs the SonarQube scanner using  an Invoke Gradle script step. Note that the new SonarQube plugin does not require any property definition in the build.gradle file. But the plugin does require a system property to be set that provides the path to the output file of the check-key-measures operation run in the previous step. This property is available to the Structure101 plugin when SonarQube invokes its scan.

The path must match that set in the structure101-build-config.xml file

 <operation type="check-key-measures">
<argument name="output-file" value="const(THIS_FILE)/target/structure101/key-measures.xml"/>
<argument name="output-file-for-snapshot" value="const(THIS_FILE)/s101working/structure101report/snapshot-key-measures.xml"/>
<argument name="baseline" value="14"/>
<argument name="useProjectFileSpec" value="true"/>
<argument name="useProjectFileDiagrams" value="true"/>
<argument name="fail-on-architecture-violations" value="false"/>
<argument name="fail-on-fat-package" value="false"/>
<argument name="fail-on-fat-class" value="false"/>
<argument name="fail-on-fat-method" value="false"/>
<argument name="fail-on-feedback-dependencies" value="false"/>
<argument name="fail-on-spec-violation-dependencies" value="false"/>
<argument name="fail-on-total-problem-dependencies" value="false"/>
<argument name="fail-on-spec-item-violations" value="false"/>
<argument name="fail-on-biggest-class-tangle" value="false"/>
<argument name="fail-on-tangled-package" value="false"/>
<argument name="fail-on-architecture-violations" value="false"/>
<argument name="identifier-on-violation" value="S101 key measure violation"/>
</operation>

 

The next step is a Conditional step (single) that uses Text Finder to check the console output for failure messages from the check-key-measures operation and the Set the build result step to set the build status if any are found. In this example the build is set to failed if any of the messages are detected using a simple OR’d regex.

The messages are: –

FAIL! fat0 has increased by 
FAIL! fat1 has increased by
FAIL! fat2 has increased by
FAIL! feedbackDependencies has increased by
FAIL! specViolationDependencies has increased by
FAIL! totalProblemDependencies has increased by
FAIL! specItemViolations has increased by
FAIL! biggestClassTangles has increased by
FAIL! tangledDesign has increased by
FAIL! numViolations has increased by

if desired a different build status can be set for the messages by creating multiple Text Finder steps.

 

The next step is also a single conditional that only runs if the build is successful. It executes a shell command that updates the baseline argument of the check-key-measures, check-spec and check-architecture operations in the structure101-build-config.xml file and then commits the changes.

The awk command replaces the line containing the baseline argument with the content of the baseline_tag variable. This should be identical to the variable used to set the label argument.

awk -v baseline_tag="\<argument name\=\"baseline\" value\=\"$BRANCH_NAME$BUILD_ID\"/\>" '{gsub("\<argument name=\"baseline\" value\=\".*", baseline_tag)}1' structure101-build-config.xml > baseline-structure101-build-config.xml 
mv -f baseline-structure101-build-config.xml structure101-build-config.xml 
git commit -am "Increment baseline label"

Finally the Git Publisher Post-build Action is used to push the baseline change to the repository.

 

Process Considerations

The automated update of the baseline version is critical to the success of the process. For this reason it is suggested that that updates to the structure101-build-config.xml file are carefully controlled. After the initial publish further manual publishing of snapshots to the Structure101 repository is not recommended.

Forcing Success for a Build Failing the Structure101 Checks

It may be necessary to force the success of a failing build. This can be achieved by manually updating the baseline arguments in the structure101-build-config.xml file to the label of the last failed build. The violations causing the failure will no longer be treated as new and the build will succeed.

Updating the Structure Spec or Diagrams

The process for updating (or creating new) architecture diagrams or structure spec is simply to commit the modifed java.hsp file to source control. The next Jenkins build will check out the updated file and perform the check operations using the local spec and architecture diagrams in the Structure101 project file. The properties useProjectFileSpec and useProjectFileDiagrams force this behaviour.

Potential Problems

The Structure101 Groups are not displaying in the Measures Page

The Structure101 Quality Profile is not configured for the SonarQube project. Update the project configuration.

The SonarQube scanner is not configured with the structure101.reportdir system property or it does not match the output-file property of the check-key-measures operation. Check the build log for error messages and update the build config file and Jenkins SonarQube step to use the same path.

Using the Structure101 SonarQube Plugin with Maven

The Structure101 Key Measures can be reported via SonarQube. The configuration to do this is straight forward and can re-use an existing Structure101 Build configuration file. [Build 14385 of the Structure101 SonarQube plugin]

It is assumed that SonarQube Scanner is already configured within your maven build.

The plugin is installed by copying the sonar-structure101-plugin.jar file to the SonarQube extensions/plugins folder and restarting the SonarQube daemon.

The plugin is configured by adding properties in the project’s maven configuration. The Structure101 Sensor requires the following properties:
structure101.java.classpath
structure101.java.repository
structure101.java.project
or
structure101.java.pom
structure101.java.repository
structure101.java.project
or
structure101.java.headless.config

If you have an existing Structure101 Build configuration file this can be used by the plugin requiring just the structure101.java.headless.config property to be set.

<structure101.java.headless.config>${basedir}\structure101-build-conf.xml</structure101.java.headless.config>

The structure101.java.disabled property should also be set to enable the Structure101 scanner.

<structure101.java.disabled>false</structure101.java.disabled>

The plugin will use the class path, project and repository defined via the configuration file. (see Configuring Structure101 Build for details)

If a build configuration file is not used then the repository and project must be defined.

<structure101.java.repository>c:/structure101-repository</structure101.java.repository>
<structure101.java.project>plugins</structure101.java.project>

The classpath to analyse can be defined as a pom.xml file or as a  standard Java classpath

<structure101.java.pom>${basedir}\pom.xml</structure101.java.pom>

<structure101.java.classpath>${basedir}\target/classes</structure101.java.classpath>


Structure101 Measures

The Structure101 Key Measures can be viewed in the SonarQube web application’s Measures page

Configuring Structure101 Build

Structure101 Build can be added to your existing CI processes to check for new structural violations and deteriorating key measures.

Structure101 Build is delivered as a zip archive. Installation is simply unpacking to a suitable folder location. The core jar file is structure101-java-build.jar.   A minimum Java 8 is the only requirement for running the Build process.

The build process is configured using an XML file. This is typically located in the project root folder alongside the Studio (.hsp) and Workspace (.hsw) files.
Any and all paths in these files should use const(THIS_FILE) to ensure the paths resolve correctly regardless of the root folder location.

The Build process is run with a simple command line passing the configuration file as the only argument: –

java -jar <path-to>/structure101-java-build.jar structure101-build-configuration.xml

Below is a simple example configuration file that runs the check-spec operation.

<?xml version="1.0" encoding="UTF-8"?>
<headless version="1.0">
    <operations>
        // check for structure spec violations
        <operation type="check-spec">
           <argument name="output-dir" value="const(THIS_FILE)/s101-results"/>
            <argument name="onlyNew" value=“true"/>
            <argument name="check-unassociated" value="true"/>
            <argument name="fail-on-violations" value="false"/>
            <argument name="identifier-on-violation" value="build unstable"/>
        </operation>
   </operations>
   <arguments>
       <argument name="local-project" value="const(THIS_FILE)/structure101-studio.java.hsp">
        </argument>
    </arguments>
</headless>

The operations block contains the operations to be run. They are executed in the order they appear in the file. The arguments block must contain the local-project argument. The classpath that Build will analyse can be taken from the local project file or through an override to the local-project argument.

The simplest setup (shown above) is to reference the local Studio project file from which build will take the classpath. The repository URL and project name are also taken from the local project file. [For version 5.0.14385 and earlier the project and repository arguments must be supplied]

All three can be overridden in the configuration XML. The repository URL and project name are overridden using the repository and project arguments.

<arguments>
       <argument name="local-project" value="const(THIS_FILE)/structure101-studio.java.hsp">
        </argument>
        //Override the .hsp file
        <argument name="repository" value="c:\structure101-repository"/>
        <argument name="project" value="S101 Example"/>
    </arguments>
</headless>

The classpath is overridden using the override classpath attribute. In this example classpathentry elements are being used so that module names can be defined. This form should be used if you are using a Workspace derived Studio project.

<arguments>
    <argument name="local-project" >
        <override attribute="classpath">
            <classpathentry kind="lib" path="const(THIS_FILE)/util/build/classes/java/main" module="util"/>
            <classpathentry kind="lib" path="const(THIS_FILE)/nalpeiron/build/classes/java/main" module="nalpeiron"/>
        </override>
   </argument>
   <argument name="repository" value="c:\structure101-repository"/>
   <argument name="project" value="Extract Module Example"/>
</arguments>

In the final example below the classpath override attribute value is used to define a traditional delimiter separated classpath.
Also in this example the local-project value is omitted. In this case the classpath override, repository and project arguments are mandatory.

<arguments>
    <argument name="local-project">
        <override attribute="classpath" value="C:\classes-to-parse;c:\more-classes-to-parse"/>
    </argument>
    <argument name="repository" value="C:\repository"/>
    <argument name="project" value="my-project"/>
</arguments>

Using the Structure101 SonarQube Plugin with Gradle

The Structure101 Key Measures can be reported via SonarQube. The configuration to do this is straight forward and can re-use an existing Structure101 Build configuration file. [Build 14385 of the Structure101 SonarQube plugin]

It is assumed that SonarQube Scanner is already configured within your gradle build.

The plugin is installed by copying the sonar-structure101-plugin.jar file to the SonarQube extensions/plugins folder and restarting the SonarQube daemon.

The plugin is configured by adding to the existing SonarQube properties in the project’s gradle configuration.

If you have an existing Structure101 Build configuration file this can be used by the plugin.

sonarqube {
    properties {
        property "sonar.projectName", "Example Project"
        property "sonar.projectKey", "$project.group:$project.name"
        property "sonar.language", "java"
        <snip>
        property "structure101.java.disabled", "false"
        property "structure101.java.headless.config", "$rootDir/structure101-build-conf.xml"
    }
}

The plugin will use the class path, Structure101 Repository location and Structure101 project name defined in the configuration file. The Structure Spec and Architecture Diagrams in the Repository project are used during the plugin analysis in the same way they are used in the Structure101 Build process.

 

Note that defining the structure101.java.classpath property does not allow each class path entry to be associated with a module name. If you are using Structure101 Workspace and Studio projects created from a Workspace .hsw file it is recommended that you use a Build configuration file. The class path defined in the configuration file allows each entry to be associated with a module name.

This class path is either an override defined in the configuration file or the class path of the Studio project referenced in the configuration file. (See this post for details)

Where there is no existing Structure101 Build configuration file and the module naming is not required then the classpath, project and repository location properties should be used.

sonarqube {
    properties {
        property "sonar.projectName", "Example Project"
        property "sonar.projectKey", "$project.group:$project.name"
        property "sonar.language", "java"
        <snip>
        property "structure101.java.disabled", "false"
        property "structure101.java.project", "My Application"
        property "structure101.java.repository", "c:/my-repository"
        property "structure101.java.classpath", "$rootDir/project1/build/classes/java/main/;$rootDir/project2/...”
    }
}

Turn Off Sub-Projects in Multi-Project Builds

It is not necessary to run the Structure101 analysis on the individual sub-projects. It should be disabled using the structure101.java.disabled property. This will save unnecessary processing during  your build.

subprojects {
    sonarqube {
        properties {
            property "structure101.java.disabled", true
        }
    }
}

Structure101 Measures

The Structure101 Key Measures can be viewed in the SonarQube web application’s Measures page

Structure101 Build Configuration with Workspace based Studio Projects

Structural refactoring initiatives typically make use of the synchronisation that is possible between Studio and Workspace.  Careful configuration of Structure101 Build will ensure the Key Measures can be tracked consistently across Studio, Workspace, SonarQube and the Structure101 Web Application.

Workspace and Studio

The synchronisation between Studio and Workspace begins by creating the Studio project from the Workspace .hsw file. This ensures the classpaths used in Workspace are also used by Studio. Whenever the Studio project is opened the .hsw file is checked for changes to the modules and their classpaths and the Studio project is updated accordingly.

The Studio project is used to create refactoring Action Lists and Structure Specs. These are shared with the development team by publishing a shared Action List and Structure Spec to the Structure101 Repository.

 

In the Workspace settings the published project is referenced.

Build

Structure101 Build needs the classpaths it should process, a project name and its repository location. The repository project has the snapshot against which comparisons will be made. It will also contain the Structure Spec against which the model will be validated. This information can be captured in a structure101-build-config.xml file.

The config file can reference the Studio .hsp project file from which the repository, project name and classpath can be retrieved. Any of them can be overridden in the configuration file.

 

Web Application

The Structure101 Web Application is configured to use the repository. In addition to viewing the projects that have been published it can be used for remote access to the Repository.

Source Control of the Workspace, Studio and Build Files

The Studio .hsp project file, Workspace .hsw file and build configuration file should be kept under source control in the root folder of the source tree. All file paths in all three files should begin with or be relative to const(THIS_FILE).  So that the source tree can be checked out into any folder path and the file and classpath references in the three files will resolve to the correct folders.

It is recommended that the commit privileges on these files are restricted to the team leads/developers responsible for the restructuring initiative.

 

The integration of all the Structure101 components is shown below.

 

Merging Structure101 Studio Action Lists

The action lists created in Structure101 Studio are saved in the project .hsp file. Currently there is no ‘Merge Action List’ feature in Studio. But the lists can be merged by editing the .hsp XML file.

To avoid overwriting your changes close the project in Studio before editing the .hsp file.

The structure of the Action List element is: –

  <restructuring>
    <set version="3" name="prodconf_v2" hiview="Codemap" active="true" todo="false" list="0">
      <namemap applyMappings="true">
        <map from="...pstnipvs.nces.CallBarringError" to="...pstnipvs.nces.CallBarringError" />
	…
      </namemap>
      <action>
	…
      </action>
      <action>
	…
      </action>
	…
    </set>

Merging action lists is achieved by appending the <map> elements and the <action> elements of one list to another.

When merging multiple lists you must append the maps and actions of each list in turn to maintain the overall order.

The merge may produce unexpected results unless each class or package is mapped in just one of the action lists being merged.

Structure101 Workspace IntelliJ IDEA and Gradle

The IDEA gradle import creates module compile output paths that are different to the gradle build output paths. Structure101 Workspace uses the IDEA module compile output paths to find the bytecode. So, running a gradle build will not propagate changes to the code into the Structure Map.

A workaround is to run the IDEA build to generate bytecode into the module compile output paths. But this only works if the setting “Delegate IDE build/run actions to gradle” (under Project Settings | Build, Execution, Deployment | Build Tools | Gradle | Runner) is unchecked. And it is not ideal since there are two sets of bytecode generated.

An alternative solution is to use the gradle Idea plugin to override the module compile output paths when the project is imported into IDEA.

Add the following to the root build.gradle file: –
(Use directory names appropriate for your gradle build)

apply plugin: 'idea'

subprojects {
  idea {
   module {
     outputDir file("$buildDir/classes/java/main")
     testOutputDir file("$buildDir/classes/java/test")
     inheritOutputDirs = false
  }
 }
}


Import your gradle project into IDEA in the usual way. You can use either the file or folder based format for the project

If you check “Create separate module per source root” in the import dialog the import will create IDEA modules for each source root in the  gradle module. The module names will be suffixed, usually with _main and _test. Having separate modules for test code is useful when extracting modules from the monolith. (After the import completes you can toggle this setting off and on in the gradle settings dialog)

The separate test modules allow the test code to be excluded from the Structure Map by adding excludes in the Structure101 Workspace settings dialog (<module_name>.*). This patterns excludes the code and the module from the Structure Map.

After import the module compile output paths will be set to the gradle output paths that you added to the build.gradle file. Building with gradle at the command line will now update the bytecode that Workspace is referencing.

If you wish to trigger the gradle build using the IDEA build commands the “Delegate IDE build/run actions to gradle” setting can be checked.

Update:

The latest version of the Workspace plugin for IDEA now has an additional option on the settings page.

Include ‘Test Source Folders’ when populating Structure Map

This option is off by default. Any code marked as test in the IDEA module settings Sources tab (see below) will not be included in the Structure Map model if this new option is unchecked. When checked, additional top level modules will be created in the Structure Map containing test code. These modules will have the same name as the main module with the configurable suffix (-test by default).

Note that a Studio project created from the Workspace .hsw file will show the same modules as Workspace. So if the Test option is checked then Studio will also show the test modules.