Blockchain – an introduction

The blockchain concept is a fundamental component of various crypto payment methods such as Bitcoin and Ethereum. But what exactly is blockchain technology, and what other applications does this concept have? Essentially, blockchain is structured like a backward-linked list. Each element in the list points to its predecessor. So, what makes blockchain so special?

Blockchain extends the list concept by adding various constraints. One of these constraints is ensuring that no element in the list can be altered or removed. This is relatively easy to achieve using a hash function. We encode the content of each element in the list into a hash using a hash algorithm. A wide range of hash functions are now available, with SHA-512 being a current standard. This hash algorithm is already implemented in the standard library of almost every programming language and can be used easily. Specifically, this means that the SHA-512 hash is generated from all the data in a block. This hash is always unique and never occurs again. Thus, the hash serves as an identifier (ID) to locate a block. An entry in the block is a reference to its predecessors. This reference is the hash value of the predecessor, i.e., its ID. When implementing a blockchain, it is essential that the hash value of the predecessor is included in the calculation of the hash value of the current block. This detail ensures that elements in the blockchain can only be modified with great difficulty. Essentially, to manipulate the element one wishes to alter, all subsequent elements must also be changed. In a large blockchain with a very large number of blocks, such an undertaking entails an enormous computational effort that is very difficult, if not impossible, to accomplish.

This chaining provides us with a complete transaction history. This also explains why crypto payment methods are not anonymous. Even though the effort required to uniquely identify a transaction participant can be enormous, if this participant also uses various obfuscation methods with different wallets that are not linked by other transactions, the effort increases exponentially.

Of course, the mechanism just described still has significant weaknesses. Transactions, i.e., the addition of new blocks, can only be considered verified and secure once enough successors have been added to the blockchain to ensure that changes are more difficult to implement. For Bitcoin and similar cryptocurrencies, a transaction is considered secure if there are five subsequent transactions.

To avoid having just one entity storing the transaction history—that is, all the blocks of the blockchain—a decentralized approach comes into play. This means there is no central server acting as an intermediary. Such a central server could be manipulated by its operator. With sufficient computing power, this would allow for the rebuilding of even very large blockchains. In the context of cryptocurrencies, this is referred to as chain reorganization. This is also the criticism leveled at many cryptocurrencies. Apart from Bitcoin, no other decentralized and independent cryptocurrency exists. If the blockchain, with all its contained elements, is made public and each user has their own instance of this blockchain locally on their computer, where they can add elements that are then synchronized with all other instances of the blockchain, then we have a decentralized approach.

The technology for decentralized communication without an intermediary is called peer-to-peer (P2P). P2P networks are particularly vulnerable in their early stages, when there are only a few participants. With a great deal of computing power, one could easily create a large number of so-called “Zomi peers” that influence the network’s behavior. Especially in times when cloud computing, with providers like AWS and Google Cloud Platform, can provide virtually unlimited resources for relatively little money, this is a significant problem. This point should not be overlooked, particularly when there is a high financial incentive for fraudsters.

There are also various competing concepts within P2P. To implement a stable and secure blockchain, it is necessary to use only solutions that do not require supporting backbone servers. The goal is to prevent the establishment of a master chain. Therefore, questions must be answered regarding how individual peers can find each other and which protocol they use to synchronize their data. By protocol, we mean a set of rules, a fixed framework for how interaction between peers is regulated. Since this point is already quite extensive, I refer you to my 2022 presentation for an introduction to the topic.

Another feature of blockchain blocks is that their validity can be easily and quickly verified. This simply requires generating the SHA-512 hash of the entire contents of a block. If this hash matches the block’s ID, the block is valid. Time-sensitive or time-critical transactions, such as those relevant to payment systems, can also be created with minimal effort. No complex time servers are needed as intermediaries. Each block is appended with a timestamp. This timestamp must, however, take into account the location where it is created, i.e., specify the time zone. To obscure the location of the transaction participants, all times in the different time zones can be converted to the current UTC 0.

To ensure that the time is correctly set on the system, a time server can be queried for the current time when the software starts, and a correction message can be displayed if there are any discrepancies.

Of course, time-critical transactions are subject to a number of challenges. It must be ensured that a transaction was carried out within a defined time window. This is a problem that so-called real-time systems have to deal with. The double-spending problem also needs to be prevented—that is, the same amount being sent twice to different recipients. In a decentralized network, this requires confirmation of the transaction by multiple participants. Classic race conditions can also pose a problem. Race conditions can be managed by applying the Immutable design pattern to the block elements.

To prevent the blockchain from being disrupted by spam attacks, we need a solution that makes creating a single block expensive. We achieve this by incorporating computing power. The participant creating a block must solve a puzzle that requires a certain amount of computing time. If a spammer wants to flood the network with many blocks, their computing power increases exorbitantly, making it impossible for them to generate an unlimited number of blocks in a short time. This cryptographic puzzle is called a nonce, which stands for “number used only once.” The nonce mechanism in the blockchain is also often referred to as Proof of Work (POW) and is used in Bitcoin to verify the blocks by the miners.

The nonce is a (pseudo)random number for which a hash must be generated. This hash must then meet certain criteria. These could be, for example, two or three leading zeros in the hash. To prevent arbitrary hashes from being inserted into the block, the random number that solves the puzzle is stored directly. A nonce that has already been used cannot be used again, as this would circumvent the puzzle. When generating the hash from the nonce, it must meet the requirements, such as leading zeros, to be accepted.

Since finding a valid nonce becomes increasingly difficult as the number of blocks in a blockchain grows, it is necessary to change the rules for such a nonce cyclically, for example, every 2048 blocks. This also means that the rules for a valid nonce must be assigned to the corresponding blocks. Such a set of rules for the nonce can easily be formulated using a regular expression (regex).

We’ve now learned a considerable amount about the ruleset for a blockchain. So it’s time to consider performance. If we were to simply store all the individual blocks of the blockchain in a list, we would quickly run out of memory. While it’s possible to store the blocks in a local database, this would negatively impact the blockchain’s speed, even with an embedded solution like SQLite. A simple solution would be to divide the blockchain into equal parts, called chunks. A chunk would have a fixed length of 2048 valid blocks, and the first block of a new chunk would point to the last block of the previous chunk. Each chunk could also contain a central rule for the nonce and store metadata such as minimum and maximum timestamps.

To briefly recap our current understanding of the blockchain ruleset, we’re looking at three different levels. The largest level is the blockchain itself, which contains fundamental metadata and configurations. Such configurations include the hash algorithm used. The second level consists of so-called chunks, which contain a defined set of block elements. As mentioned earlier, chunks also contain metadata and configurations. The smallest element of the blockchain is the block itself, which comprises an ID, the described additional information such as a timestamp and nonce, and the payload. The payload is a general term for any data object that is to be made verifiable by the blockchain. For Bitcoin and other cryptocurrencies, the payload is the information about the amount being transferred from Wallet A (source) to Wallet B (destination).

Blockchain technology is also suitable for many other application scenarios. For example, the hash values ​​of open-source software artifacts could be stored in a blockchain. This would allow users to download binary files from untrusted sources and verify them against the corresponding blockchain. The same principle could be applied to the signatures of antivirus programs. Applications and other documents could also be transmitted securely in governmental settings. The blockchain would function as a kind of “postal stamp.” Accounting, including all receipts for goods and services purchased and sold, is another conceivable application.

Depending on the use case, an extension of the blockchain would be the unique signing of a block by its creator. This would utilize the classic PKI (Public Key Infrastructure) method with public and private keys. The signer stores their public key in the block and creates a signature using their private key via the payload, which is then also stored in the block.

Currently, there are two freely available blockchain implementations: BitcoinJ and Web3j for Ethereum. Of course, it’s possible to create your own universally applicable blockchain implementation using the principles just described. The pitfalls, naturally, lie in the details, some of which I’ve already touched upon in this article. Fundamentally, however, blockchain isn’t rocket science and is quite manageable for experienced developers. Anyone considering trying their hand at their own implementation now has sufficient basic knowledge to delve deeper into the necessary details of the various technologies involved.


Count on me – Java Enums

As an experienced developer, you often think that core Java topics are more for beginners. But that doesn’t necessarily have to be the case. One reason is, of course, the tunnel vision that develops in ‘pros’ over time. You can only counteract this tunnel vision by occasionally revisiting seemingly familiar topics. Unfortunately, enumerations in Java are somewhat neglected and underutilized. One possible reason for this neglect could be the many tutorials on enums available online, which only demonstrate a trivial use case. So, let’s start by looking at where enums are used in Java.

Let’s consider a few simple scenarios we encounter in everyday development. Imagine we need to implement a hash function that can support multiple hash algorithms like MD5 and SHA-1. At the user level, we often have a function that might look something like this: String hash(String text, String algorithm); – a technically correct solution.

Another example is logging. To write a log entry, the method could look like this: void log(String message, String logLevel); and examples in this style can be continued indefinitely.

The problem with this approach is free parameters like logLevel and algorithm, which allow for only a very limited number of possibilities for correct use. The risk of developers filling these parameters incorrectly under project pressure is quite high. Typos, variations in capitalization, and so on are potential sources of error. There are certainly a number of such design decisions in the Java API that could be improved.

As is often the case, there are many ways to Rome when it comes to improving the use of the API, i.e., the functions used externally. One approach that is quite common is the use of constants. To illustrate the solution a little better, I’ll take the example of hash functionality. The two hash algorithms MD5 and SHA-1 should be supported in the Hash class.

void Hash {
	
	public final static MD5 = "1";
	public final static SHA-1 = "2";

	/**
	 * Creates a hash from text by using the chosen algorithm.
	 */
	public String calculateHash(String text, String algorithm) {
		String hash = "";

		if(algorithm.equals(MD5)) {
			hash = MD5.hash(text);
		}
		if(algorithm.equals(SHA-1)) {
			hash = SHA-1.hash(text);
		}
		return hash;
	}
}

// Call
calculateHash("myHashValue", Hash.MD5);

In the Hash class, the two constants MD5 and SHA-1 are defined to indicate which values ​​are valid for the parameter algorithm. Developers with some experience recognize this configuration quite quickly and use it correctly. Technically, the solution is absolutely correct. However, even if something is syntactically correct, it doesn’t necessarily mean it’s an optimal solution in the semantic context. The strategy presented here disregards the design principles of object-oriented programming. Although we already provide valid input values ​​for the parameter algorithm through the constants, there’s no mechanism that enforces their use.

One might argue that this isn’t important as long as the method is called with a valid parameter. But that’s not the case. Every call to the calculateHash method without using the constants will lead to potential problems during refactoring. For example, if you want to change the value “2” to SHA-1 for consistency, this will lead to errors wherever the constants aren’t used. So, how can we improve this? The preferred solution should be the use of enums.

All enums defined in Java are derived from the class java.lang.Enum [2]. A selection of the methods is:

  • name() returns the name of this enum constant exactly as it appears in its enum declaration.
  • ordinal() returns the ordinal number of this enum constant (its position in the enum declaration, with the first constant assigned ordinal number zero).
  • toString() returns the name of this enum constant exactly as it appears in the declaration.

Dies ist die einfachste Variante, wie man ein Enum definieren kann. Eine einfache Aufzählung der gültigen Werte als Konstanten. In dieser Notation sind Konstantenname und Konstantenwert identisch. Wichtig beim Umgang mit Enums ist sich bewusst zu sein das die Namenkonventionen für Konstanten gelten.

The names of variables declared as class constants and of ANSI constants should be written exclusively in uppercase letters, with underscores (“_”) separating the words. (ANSI constants should be avoided to simplify debugging.) Example: static final int GET_THE_CPU = 1;

[1] Oracle Documentation

This property leads us to the first problem with the simplified definition of enums. If, for whatever reason, we want to define SHA-1 instead of SHA-1, this will result in an error, since “-” does not conform to the convention for constants. Therefore, the following construct is most commonly used for defining enums:

public enum HashAlgorithm {

    MD5("MD5"),
    SHA1("SHA-1");

    private String value;

    HashAlgorithm(final String value) {
        this.value = value;
    }

    public String getValue() {
        return this.value;
    }
}

In this example, the constant SHA1 is extended with the string value “SHA-1”. To access this value, it’s necessary to implement a way to retrieve it. This access is provided via the variable value, which has the corresponding getter getValue(). To populate the variable value, we also need a constructor. The following test case demonstrates the different ways to access the entry SHA1 and their corresponding outputs.

public class HashAlgorithmTest {

    @Test
    void enumNames() {
        assertEquals("SHA1",
                HashAlgorithm.SHA1.name());
    }

    @Test
    void enumToString() {
        assertEquals("SHA1",
                HashAlgorithm.SHA1.toString());
    }
    
    @Test
    void enumValues() {
        assertEquals("SHA-1",
                HashAlgorithm.SHA1.getValue());
    }


    @Test
    void enumOrdinals() {
        assertEquals(1,
                HashAlgorithm.SHA1.ordinal());
    }
}

First, we access the name of the constant, which in our example is SHA1. We obtain the same result using the toString() method. We retrieve the content of SHA1, i.e., the value SHA-1, by calling our defined getValue() method. Additionally, we can query the position of SHA1 within the enum. In our example, this is the second position where SHA1 occurs, and therefore has the value 1. Let’s look at the change for enums in the implementation of the hash class from Listing 1.

public String calculateHash(String text, HashAlgorithm algorithm) {
    String hash = "";
    switch (algorithm) {
        case MD5:
		hash = MD5.hash(text);
            break;
        case SHA1:
		hash = SHA1.hash(text);
            break;
    }
    return hash;
}

//CaLL
calculateHash("text2hash", HashAlgorithm.MD5);

In this example, we see a significant simplification of the implementation and also achieve a secure use for the algorithm parameter, which is now determined by the enum. Certainly, there are some critics who might complain about the amount of code to be written and point out that it could be done much more compactly. Such arguments tend to come from people who have no experience with large projects involving millions of lines of source code or who don’t work in a team with multiple programmers. Object-oriented design isn’t about avoiding every unnecessary character, but about writing readable, fault-tolerant, expressive, and easily modifiable code. We have covered all these attributes in this example using enums.

According to Oracle’s tutorial for Java enums [3], much more complex constructs are possible. Enums behave like Java classes and can also be enriched with logic. The specification primarily provides the compare() and equals() methods, which enable comparison and sorting. For the hash algorithm example, this functionality would not add any value. It’s also generally advisable to avoid placing additional functionality or logic within enums, as these should be treated similarly to pure data classes.

The fact that enums are indeed an important topic in Java development is demonstrated by the fact that J. Bloch dedicated an entire chapter of nearly 30 pages to them in his book “Effective Java.”

With this, we have worked through a comprehensive example of the correct use of enums and, above all, learned where this language construct finds meaningful application in Java.

Resources

Abonnement / Subscription

[English] This content is only available to subscribers.

[Deutsch] Diese Inhalte sind nur für Abonnenten verfügbar.


PDF generation with TP-CORE

PDF is arguably the most important interoperable document format between operating systems and devices such as computers, smartphones, and tablets. The most important characteristic of PDF is the immutability of the documents. Of course, there are limits, and nowadays, pages can easily be removed or added to a PDF document using appropriate programs. However, the content of individual pages cannot be easily modified. Where years ago expensive specialized software such as Adobe Acrobat was required to work with PDFs, many free solutions are now available, even for commercial use. For example, the common office suites support exporting documents as PDFs.

Especially in the business world, PDF is an excellent choice for generating invoices or reports. This article focuses on this topic. I describe how simple PDF documents can be used for invoicing, etc. Complex layouts, such as those used for magazines and journals, are not covered in this article.

Technically, the freely available library openPDF is used. The concept is to generate a PDF from valid HTML code contained in a template. Since we’re working in the Java environment, Velocity is the template engine of choice. TP-CORE utilizes all these dependencies in its PdfRenderer functionality, which is implemented as a facade. This is intended to encapsulate the complexity of PDF generation within the functionality and facilitate library replacement.

While I’m not a PDF generation specialist, I’ve gained considerable experience over the years, particularly with this functionality, in terms of software project maintainability. I even gave a conference presentation on this topic. Many years ago, when I decided to support PDF in TP-CORE, the only available library was iText, which at that time was freely available with version 5. Similar to Linus Torvalds with his original source control management system, I found myself in a similar situation. iText became commercial, and I needed a new solution. Well, I’m not Linus, who can just whip up Git overnight, so after some waiting, I discovered openPDF. OpenPDF is a fork of iText 5. Now I had to adapt my existing code accordingly. This took some time, but thanks to my encapsulation, it was a manageable task. However, during this adaptation process, I discovered problems that were already causing me difficulties in my small environment, so I released TP-CORE version 3.0 to achieve functional stability. Therefore, anyone using TP-CORE version 2.x will still find iText5 as the PDF solution. But that’s enough about the development history. Let’s look at how we can currently generate PDFs with TP-CORE 3.1. Here, too, my main goal is to achieve the highest possible compatibility.

Before we can begin, we need to include TP-CORE as a dependency in our Java project. This example demonstrates the use of Maven as the build tool, but it can easily be switched to Gradle.

<dependency>
    <groupId>io.github.together.modules</groupId>
    <artifactId>core</artifactId>
    <version>3.1.0</version>
</dependency>

To generate an invoice, for example, we need several steps. First, we need an HTML template, which we create using Velocity. The template can, of course, contain placeholders for names and addresses, enabling mail merge and batch processing. The following code demonstrates how to work with this.

<h1>HTML to PDF Velocity Template</h1>

<p>Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet.</p>

<p>Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.</p>

<h2>Lists</h2>
<ul>
    <li><b>bold</b></li>
    <li><i>italic</i></li>
    <li><u>underline</u></li>
</ul>

<ol>
    <li>Item</li>
    <li>Item</li>
</ol>

<h2>Table 100%</h2>
<table border="1" style="width: 100%; border: 1px solid black;">
    <tr>
        <td>Property 1</td>
        <td>$prop_01</td>
        <td>Lorem ipsum dolor sit amet,</td>
    </tr>
    <tr>
        <td>Property 2</td>
        <td>$prop_02</td>
        <td>Lorem ipsum dolor sit amet,</td>
    </tr>
    <tr>
        <td>Property 3</td>
        <td>$prop_03</td>
        <td>Lorem ipsum dolor sit amet,</td>
    </tr>
</table>

<img src="path/to/myImage.png" />

<h2>Links</h2>
<p>here we try a simple <a href="https://together-platform.org/tp-core/">link</a></p>

template.vm

The template contains three properties: $prop_01, $prop_02, and $prop_03, which we need to populate with values. We can do this with a simple HashMap, which we insert using the TemplateRenderer, as the following example shows.

Map<String, String> properties = new HashMap<>();
properties.put("prop_01", "value 01");
properties.put("prop_02", "value 02");
properties.put("prop_03", "value 03");

TemplateRenderer templateEngine = new VelocityRenderer();
String directory = Constraints.SYSTEM_USER_HOME_DIR + "/";
String htmlTemplate = templateEngine
                .loadContentByFileResource(directory, "template.vm", properties);

The TemplateRenderer requires three arguments:

  • directory – The full path where the template is located.
  • template – The name of the Velocity template.
  • properties – The HashMap containing the variables to be replaced.

The result is valid HTML code, from which we can now generate the PDF using the PdfRenderer. A special feature is the constant SYSTEM_USER_HOME_DIR, which points to the user directory of the currently logged-in user and handles differences between operating systems such as Linux and Windows.

PdfRenderer pdf = new OpenPdfRenderer();
pdf.setTitle("Title of my own PDF");
pdf.setAuthor("Elmar Dott");
pdf.setSubject("A little description about the PDF document.");
pdf.setKeywords("pdf, html, openPDF");
pdf.renderDocumentFromHtml(directory + "myOwn.pdf", htmlTemplate);

The code for creating the PDF is clear and quite self-explanatory. It’s also possible to hardcode the HTML. Using the template allows for a separation of code and design, which also makes later adjustments flexible.

The default format is A4 with the dimensions size: 21cm 29.7cm; margin: 20mm 20mm; and is defined as inline CSS. This value can be customized using the setFormat(String format); method. The first value represents the width and the second value the height.

The PDF renderer also allows you to delete individual pages from a document.

PdfRenderer pdf = new OpenPdfRenderer();
PdfDocument original = pdf.loadDocument( new File("document.pdf") );
PdfDocument reduced = pdf.removePage(original, 1, 5);
pdf.writeDocument(reduced, "reduced.pdf");

We can see, therefore, that great emphasis was placed on ease of use when implementing the PDF functionality. Nevertheless, there are many ways to create customized PDF documents. This makes the solution of defining the layout via HTML particularly appealing. Despite the PdfRenderer’s relatively limited functionality, Java’s inheritance mechanism makes it very easy to extend the interface and its implementation with custom solutions. This possibility also exists for all other implementations in TP-CORE.

TP-CORE is also free for commercial use and has no restrictions thanks to the Apache 2 license. The source code can be found at https://git.elmar-dott.com and the documentation, including security and test coverage, can be found at https://together-platform.org/tp-core/.

BugChaser – The limits of test coverage

The paradigms now established in software engineering, such as Test-Driven Development (TDD) and Behavior-Driven Development (BDD), along with correspondingly easy-to-use tools, have opened up a new, pragmatic perspective on the topic of software testing. Automated tests are a crucial factor in commercial software projects. Therefore, in this context, a successful testing strategy is one in which test execution proceeds without human intervention.

Test automation forms the basis for achieving stability and reducing risk in critical tasks. Such critical tasks include, in particular, refactoring, maintenance, and bug fixes. All these activities share one common goal: preventing new errors from creeping into the code.

In his 1972 article “The Humble Programmer,” Edsger W. Dijkstra stated the following:

„Program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.“

Therefore, simply automating test execution is not sufficient to ensure that changes to the codebase do not have unintended effects on existing functionality. For this reason, the quality of the test cases must be evaluated. Proven tools already exist for this purpose.

Before we delve deeper into the topic, let’s first consider what automated testing actually means. This question is quite easy to answer. Almost every programming language has a corresponding unit test framework. Unit tests call a method with various parameters and compare the return value with an expected value. If both values ​​match, the test is considered passed. Additionally, it can also be checked whether an exception was thrown.

If a method has no return value or does not throw an error, it cannot be tested directly. Methods marked as private, or inner classes, are also not easily testable, as they cannot be called directly. These must be tested indirectly through public methods that call the ‘hidden’ methods.

When dealing with methods marked as private, it is not an option to access and test the functionality they represent using techniques such as the Reflection API. We must be aware that such methods are often also used to encapsulate code fragments to avoid duplication.

public boolean method() {
    boolean success = false;
    List collector = new ArryList();
    collector.add(1);
    collector.add(2);
    collector.add(3);
    
    sortAsc(collector);
    if(collector.getFirst().equals(1)) {
        success = true;
    }
    return success;
}

private void sortAsc(List collection) {
    collection.sort( 
            (a, b) -> { 
                return -1 * a.compareTo(b); 
            });
}

Therefore, to effectively write automated tests, it is necessary to follow a certain coding style. The preceding Listing 1 simply demonstrates what testable code can look like.

Since developers also write the corresponding component tests for their own implementations, the problem of difficult-to-test code is largely eliminated in projects that follow a test-driven approach. The motivation to test now lies with the developer, as this paradigm allows them to determine whether their implementation behaves as intended. However, we must ask ourselves: Is that all we need to do to develop good and stable software?

As we might expect with such questions, the answer is no. An essential tool for evaluating the quality of tests is achieving the highest possible test coverage. A ​​distinction is made between branch and line coverage. To illustrate the difference more clearly, let’s briefly look at the pseudocode in Listing 2.

if( Expression-A OR Expression-B ) {
print(‘allow‘);
} else {
print(‘decline‘);
}

Our goal is to execute every line of code if possible. To achieve this, we already need two separate test cases: one for entering the IF branch and one for entering the ELSE branch. However, to achieve 100% branch coverage, we must cover all variations of the IF branch. In this example, that means one test that makes Expression A true, and another test that makes Expression B true. This results in a total of three different test cases.

The screenshot from the TP-CORE project shows what such test coverage can look like in ‘real-world’ projects.

Of course, this example is very simple, and in real life, there are often constructs where, despite all efforts, it’s impossible to reach all lines or branches. Exceptions from third-party libraries that need to be caught but cannot be triggered under normal circumstances are a typical example.

For this reason, while we strive to achieve the highest possible test coverage and naturally aim for 100%, there are many cases where this is not feasible. However, a test coverage of 90% is quite achievable. The industry standard for commercial projects is 85% test coverage. Based on these observations, we can say that test coverage correlates with the testability of an application. This means that test coverage is a suitable measure of testability.

However, it must also be acknowledged that the test coverage metric has its limitations. Regular expressions and annotations for data validation are just a few simple examples of where test coverage alone is not a sufficient indicator of quality.

Without going too much into the implementation details, let’s imagine we had to write a regular expression to validate input against a correct 24-hour time format. If we don’t keep the correct interval in mind, our regular expression might be incorrect. The correct interval for the 24-hour format is 00:00 – 23:59. Examples of invalid values ​​are 24:00 or 23:60. If we are unaware of this fact, errors can remain hidden in our application despite test cases, only to surface and cause problems when the application is actually used.

„… In a few cases, participants were unable to think of alternative solutions …“

The question here was whether error correction always represents the optimal solution. Beyond that, it would be necessary to clarify what constitutes an optimal solution in commercial software development projects. The statement that there are cases in which developers only know or understand one way of doing things is very illustrative. This is also reflected in our example of regular expressions (RegEx). Software development is a thought process that cannot be accelerated. Our thinking is determined by our imagination, which in turn is influenced by our experience.

This already shows us another example of sources of errors in test cases. A classic example is incorrect comparisons in collections, such as comparing arrays. The problem we are dealing with here is how variables are accessed: call by value or call by reference. With arrays, access is via call by reference, i.e., directly to the memory location. If you now assign an array to a new variable and compare both variables, they will always be the same because you are comparing the array with itself. This is an example of a test case that is essentially meaningless. However, if the implementation is correct, this faulty test case will never cause any problems.

This realization shows us that blindly striving for complete test coverage is not conducive to quality. Of course, it’s understandable that this metric is highly valued by management. However, we have also been able to demonstrate that one cannot rely on it alone. We therefore see that there is also a need for code inspections and refactorings for test cases. Since it’s impossible to read and understand all the code from beginning to end due to time constraints, it’s important to focus on problematic areas. But how can we find these problem areas? A relatively new technique helps us here. The theoretical work on this is already somewhat older; it just took a while for corresponding implementations to become available.

Cryptography – more than just coincidence

In everyday language, we use the word “coincidence” rather unreflectively. Phrases like, “I happened to be passing by here” or “What a coincidence to meet you here” are familiar to everyone. But what do we mean by that? What we’re actually trying to say is that we didn’t expect the current situation.

Coincidence is actually a mathematical term that we’ve adopted into everyday language. Coincidence means something unpredictable. Things like the exact location of any electron in an atom at a given moment. While the path I take to reach a particular destination can be arbitrary, preferences can be derived from probabilities, which then make the choice quite predictable.

Circumstances for such a scenario can be distance, personal well-being (time pressure, discomfort, or boredom), or external circumstances (weather: sunshine, rain). If I’m bored AND the sun is shining, I choose an unknown route for a bit of distraction and curiosity. If I’m short on time AND it’s raining, I choose the shortest route I know, or a route that’s as sheltered as possible. This means that the better you know a person’s habits, the more predictable their decisions are. But predictability contradicts the concept of chance.

It’s nothing new that mathematical terms with very strict definitions are temporarily adopted into our everyday language as a fad. I’d like to briefly address a very popular example, one already cited by Joseph Weizenbaum: the term chaos. In mathematical terms, chaos actually describes the fact that a very small change over very long distances significantly distorts the result, so that it can’t even be used as an estimate or approximation. A typical application is astronomy. If I point a laser beam from Earth to the moon, a deviation of just a few millimeters causes the laser beam to miss the moon by kilometers. To explain such facts to a broader audience in popular science, an association was used that if a butterfly flaps its wings in Tokyo, it can cause a storm in Berlin. Unfortunately, there are quite a few pseudoscientists who seize on this image and sell it to their peers as fact. This is, of course, nonsense. The flapping of a butterfly’s wings cannot create a storm on the other side of the globe. Just think of the impact this would have on our world, just all the birds that take to the air every day.

“Why did the mathematician’s marriage fail? His wife was unpredictable.”

But why is randomness so important in mathematics? Specifically, it’s about the broad topic of cryptography. If we choose combinations for encryption that are easy to guess, the protection is quickly lost. Here’s a small example.

Internet pages are stateless. This means that after a website is accessed and a link is clicked to go to the next page, all information from the previous page is lost. To still be able to provide things like an online shop, a shopping cart, and all the other necessary shopping functions, there is the option of storing data on the server in so-called sessions. This data often includes the user’s login. To distinguish between sessions, they have an identification (ID). The programmer then specifies how this ID is generated. One property of these IDs is that they must be unique; no ID can occur twice.

Now, one might think of using the timestamp, including the milliseconds, to generate a hash. The hash prevents anyone from immediately recognizing that the ID is created from a timestamp. A patient hacker, with a little diligence, uncovered this secret relatively quickly. Added to that is the probability that two users could create a session at the same time, which would lead to an error.

Now, one might come up with the idea of ​​assembling the SessionID from various segments such as timestamps + usernames and other details. Although increasing complexity offers a certain degree of protection, this is not true security. Professionals have methods with manageable effort to guess these ‘avoidable’ secrets. The only real protection is the use of cryptographically secure randomness. As a segment that cannot be guessed, no matter how much effort is put into it.

Before I reveal how we can address the problem, I would like to briefly discuss the typical attack vector and the damage it causes to SessionIDs. If the SessionID has been guessed by an attacker and this session is still active, the hacker can take over this session in their browser. This process is called session hijacking or session riding. The attacker who has managed to take over an active session is logged into an online service as a foreign user with a profile that does not belong to them. This allows them to perform all the actions that a legitimate user can do. It would therefore be possible to place an order in an online shop and have the goods shipped to a different address. This is a situation that must be prevented at all costs.

There are various strategies used to prevent the theft of an active session. Each of these strategies offers a certain level of protection, but the full strength is only achieved by combining the various options, as hackers are constantly evolving and looking for new opportunities. In this short article, we will only consider the aspect of how to generate a cryptographically secure session ID.

Almost all common programming languages ​​have a random() function that generates a random number. The implementation of this random number varies. Unfortunately, these generated numbers are not as random as they should be for attackers. Therefore, developers should always avoid this simple random function. Instead, there are cryptographically secure implementations for random numbers for backend languages ​​such as PHP and Java.

For Java programs, you can use the java.security.SecureRandom class. An important feature of this class is the ability to choose from various cryptographic algorithms [1]. Additionally, the starting value can be specified using the so-called seed. To demonstrate its use, here is a short code snippet:

Abonnement / Subscription

[English] This content is only available to subscribers.

[Deutsch] Diese Inhalte sind nur für Abonnenten verfügbar.

As we can see, its use is quite simple and can be easily adapted. Generating randomness is even easier in PHP. To do so, simply call the function random_int ( $min, $max ); [2]. The interval can be specified optionally.
Thus, we see that the assumption of many people that our world is largely computable is not entirely true. In many areas of the natural sciences, there are processes that we cannot calculate. These, in turn, form the basis for generating ‘true’ randomness. For applications that require very strong protection, hardware is often used. These might be devices that measure the radioactive decay of a low-radiation isotope.

The fields of cryptography and web application security are, of course, much more extensive. This article is intended to draw attention to the necessity of this topic using a fairly simple example. In doing so, I have avoided confusing and ultimately alienating potential interested parties with complicated mathematics.

Resources

Abonnement / Subscription

[English] This content is only available to subscribers.

[Deutsch] Diese Inhalte sind nur für Abonnenten verfügbar.


Pathfinder

So that we can call console programs directly across the system without having to specify the full path, we use the so-called path variable. So we save the entire path including the executable program, the so-called executable, in this path variable so that we no longer have to specify the path including the executable on the command line. By the way, the word executable derives the file extension exe, which is common in Windows. Here we also have a significant difference between the two operating systems Windows and Linux. While Windows knows whether it is a pure ASCII text file or an executable file via the file extension such as exe or txt, Linux uses the file’s meta information to make this distinction. That’s why it’s rather unusual to use these file extensions txt and exe under Linux.

Typical use cases for setting the path variable are programming languages ​​such as Java or tools such as the Maven build tool. For example, if we downloaded Maven from the official homepage, we can unpack the program anywhere on our system. On Linux the location could be /opt/maven and on Microsoft Windows it could be C:/Program Files/Maven. In this installation directory there is a subdirectory /bin in which the executable programs are located. The executable for Maven is called mvn and in order to output the version, under Linux without the entry in the path variable the command would be as follows: /opt/maven/bin/mvn -v. So it’s a bit long, as we can certainly admit. Entering the Maven installation directory in the path shortens the entire command to mvn -v. By the way, this mechanism applies to all programs that we use as a command in the console.

Before I get to how the path variable can be adjusted under Linux and Windows, I would like to introduce another concept, the system variable. System variables are global variables that are available to us in Bash. The path variable also counts as a system variable. Another system variable is HOME, which points to the logged in user’s home directory. System variables are capitalized and words are separated with an underscore. For our example with entering the Maven Executable in the path, we can also set our own system variable. The M2_HOME convention applies to Maven and JAVA_HOME applies to Java. As a best practice, you bind the installation directory to a system variable and then use the self-defined system variable to expand the path. This approach is quite typical for system administrators who simplify their server installation using system variables. Because these system variables are global and can also be read by automation scripts.

The command line, also known as shell, bash, console and terminal, offers an easy way to output the value of the system variable with echo. Using the example of the path variable, we can immediately see the difference to Linux and Windows. Linux: echo $PATH Windows: echo %PATH%

ed@local:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/home/ed/Programs/maven/bin:/home/ed/.local/share/gem//bin:/home/ed/.local/bin:/usr/share/openjfx/lib

Let’s start with the simplest way to set the path variable. In Linux we just need to edit the hidden .bashrc file. At the end of the file we add the following lines and save the content.

export M2_HOME="/opt/maven"
export PATH=$PATH:$M2_HOME/bin

We bind the installation directory to the M2_HOME variable. We then expand the path variable to include the M2_HOME system variable with the addition of the subdirectory of executable files. This procedure is also common on Windows systems, as it allows the installation path of an application to be found and adjusted more quickly. After modifying the .bashrc file, the terminal must be restarted for the changes to take effect. This procedure ensures that the entries are not lost even after the computer is restarted.

Under Windows, the challenge is simply to find the input mask where the system variables can be set. In this article I will limit myself to the version for Windows 11. It may of course be that the way to edit the system variables has changed in a future update. There are slight variations between the individual Windows versions. The setting then applies to both the CMD and PowerShell. The screenshot below shows how to access the system settings in Windows 11.

To do this, we right-click on an empty area on the desktop and select the System entry. In the System – About submenu you will find the system settings, which open the System properties popup. In the system settings we press the Environment Variables button to get the final input mask. After making the appropriate adjustments, the console must also be restarted for the changes to take effect.

In this little help, we learned about the purpose of system variables and how to store them permanently on Linux and Windows. We can then quickly check the success of our efforts in the shell using echo by outputting the contents of the variables. And we are now one step closer to becoming an IT professional.


Successful validation of ISBN validation of ISBN numbers

Developers are regularly faced with the task of checking user input for accuracy. A considerable number of standardized data formats now exist that make such validation tasks easy to master. The International Standard Book Number, or ISBN for short, is one such data format. ISBN comes in two versions: a ten-digit and a 13-digit version. From 1970 to 2006, the ten-digit version of the ISBN was used (ISBN-10), which was replaced by the 13-digit version (ISBN-13) in January 2007. Nowadays, it is common practice for many publishers to provide both versions of the ISBN for titles. It is common knowledge that books can be uniquely identified using this number. This, of course, also means that these numbers are unique. No two different books have the same ISBN (Figure 1).

The theoretical background for determining whether a sequence of numbers is correct comes from coding theory. Therefore, if you would like to delve deeper into the mathematical background of error-detecting and error-correcting codes, we recommend the book “Coding Theory” by Ralph Hardo Schulz [1]. It teaches, for example, how error correction works on compact disks (CDs). But don’t worry, we’ll reduce the necessary mathematics to a minimum in this short workshop.

The ISBN is an error-detecting code. Therefore, we can’t automatically correct a detected error. We only know that something is wrong, but we don’t know the specific error. So let’s get a little closer to the matter.

Why exactly 13 digits were agreed upon for ISBN-13 remains speculation. At least the developers weren’t influenced by any superstition. The big secret behind validation is the determination of the residual classes [2]. The algorithms for ISBN-10 and ISBN-13 are quite similar. So let’s start with the older standard, ISBN-10, which is calculated as follows:

1x1 + 2x2 + 3x3 + 4x4 + 5x5 + 6x6 + 7x7 + 8x8 + 9x9 + 10x10 = k modulo 11

Don’t worry, you don’t have to be a SpaceX rocket engineer to understand the formula above. We’ll lift the veil of confusion with a small example for ISBN 3836278340. This results in the following calculation:

(1*3) + (2*8) + (3*3) + (4*6) + (5*2) + (6*7) + (7*8) + (8*3) + (9*4) + (10*0) = 220
220 modulo 11 = 0

The last digit of the ISBN is the check digit. In the example given, this is 0. To obtain this check digit, we multiply each digit by its value. This means that the fourth position is a 6, so we calculate 4 * 6. We repeat this for all positions and add the individual results together. This gives us the amount 220. The 220 is divided by 11 using the remainder operation modulo 11. Since 11 fits exactly 20 times into 220, there is a remainder of zero. The result of 220 modulo 11 is 0 and matches the check digit, which tells us that we have a valid ISBN-10.

However, there is one special feature to note. Sometimes the last digit of the ISBN ends with X. In this case, the X must be replaced with 10.

As you can see, the algorithm is very simple and can easily be implemented using a simple for loop.

boolean success = false;
int[] isbn;
int sum = 0;

for(i=0; i<10; i++) {
sum += i*isbn[i];
}

if(sum%11 == 0) {
success = true;
}

To keep the algorithm as simple as possible, each digit of the ISBN-10 number is stored in an integer array. Based on this preparation, it is only necessary to iterate through the array. If the sum check using the modulo 11 then returns 0, everything is fine.

To properly test the function, two test cases are required. The first test checks whether an ISBN is correctly recognized. The second test checks for so-called false positives. This provokes an expected error with an incorrect ISBN. This can be quickly accomplished by changing any digit of a valid ISBN.

Our ISBN-10 validator still has one minor flaw. Digit sequences that are shorter or longer than 10, i.e., do not conform to the expected format, could be rejected beforehand. The reason for this can be seen in the example: The last digit of the ISBN-10 is a 0 – thus, the character result is also 0. If the last digit is forgotten and a check for the correct format isn’t performed, the error won’t be detected. Something that has no effect on the algorithm, but is very helpful as feedback for user input, is to gray out the input field and disable the submit button until the correct ISBN format has been entered.

The algorithm for ISBN-13 is similarly simple.

x1 + 3x2 + x3 + 3x4 + x5 + 3x6 + x7 + 3x8 + x9 + 3x10 + x11 + 3x12 + x13 = k modulo 10

As with ISBN-10, xn represents the numerical value at the corresponding position in the ISBN-13. Here, too, the partial results are summed and divided by a modulo. The main difference is that only the even-numbered positions—positions 2, 4, 6, 8, 10, and 12—are multiplied by 3, and the result is then divided by modulo 10. As an example, we calculate the ISBN-13: 9783836278348.

9 + (3*7) + 8 + (3*3) + 8 + (3*3) + 6 + (3*2) + 7 + (3*8) + 3 + (3*4) + 8 = 130
130 modulo 10 = 0

The algorithm can also be implemented for the ISBN-13 in a simple for loop.

boolean success = false;
int[] isbn;
int sum = 0;

for(i=0; i<13; i++) {
if(i%2 == 0) {
sum += 3*isbn[i];
} else {
sum += isbn[i];
}
}

if(sum%10 == 0) {
success = true;
}

The two code examples for ISBN-10 and ISBN-13 differ primarily in the if condition. The expression i % 2 calculates the modulo value 2 for the respective iteration. If the result is 0 at this point, it means it is an even number. The corresponding value must then be multiplied by 3.

This shows how useful the modulo operation % can be for programming. To keep the implementation as compact as possible, the so-called triple operator can be used instead of the if-else condition. The expression sum += (i%2) ? isbn[i] : 3 * isbn[3] is much more compact, but also more difficult to understand.

Below you will find a fully implemented class for checking the ISBN in the programming languages: Java, PHP, and C#.

Abonnement / Subscription

[English] This content is only available to subscribers.

[Deutsch] Diese Inhalte sind nur für Abonnenten verfügbar.

While the solutions presented in the examples all share the same core approach, they differ in more than just syntactical details. The Java version, for example, offers a more comprehensive variant that distinguishes more generically between ISBN-10 and ISBN-13. This demonstrates that there are many ways to Rome. It also aims to show less experienced developers different approaches and encourage them to make their own adaptations. To simplify understanding, the source code has been enriched with comments. PHP, as an untyped language, eliminates the need to convert strings to numbers. Instead, a RegEx function is used to ensure that the entered characters are type-safe.

Lessons Learned

As you can see, verifying whether an ISBN is correct isn’t rocket science. The topic of validating user input is, of course, much broader. Other examples include credit card numbers. But regular expressions also provide valuable services in this context.

Ressourcen

  • [1] Ralph-Hardo Schulz, Codierungstheorie: Eine Einführung, 2003, ISBN 978-3-528-16419-5
  • [2] Concept of modular aritmetic on Wikipedia, https://en.wikipedia.org/wiki/Modular_arithmetic

Cook Book: Maven Source Code Samples

Our Git repository contains an extensive collection of various code examples for Apache Maven projects. Everything is clearly organized by topic.

Back to table of contents: Apache Maven Master Class

  1. Token Replacement
  2. Compiler Warnings
  3. Excecutable JAR Files
  4. Enforcments
  5. Unit & Integration Testing
  6. Multi Module Project (JAR / WAR)
  7. BOM – Bill Of Materials (Dependency Management)
  8. Running ANT Tasks
  9. License Header – Plugin
  10. OWASP
  11. Profiles
  12. Maven Wrapper
  13. Shade Ueber JAR (Plugin)
  14. Java API Documantation (JavaDoc)
  15. Java Sources & Test Case packaging into JARs
  16. Docker
  17. Assemblies
  18. Maven Reporting (site)
  19. Flatten a POM
  20. GPG Signer