When we designing data models and their corresponding tables appears sometimes Boolean as datatype. In general those flags are not really problematic. But maybe there could be a better solution for the data design. Let me give you a short example about my intention.
Assume we have to design a simple domain to store articles. Like a Blog System or any other Content Management. Beside the content of the article and the name of the author could we need a flag which tells the system if the article is visible for the public. Something like published as a Boolean. But there is also an requirement of when the article is scheduled a date for publishing. In the most database designs I observed for those circumstances a Boolean: published and a Date: publishingDate. In my opinion this design is a bit redundant and also error prone. As a fast conclusion I would like to advice you to use from the beginning just Date instead of Boolean. The scenario I described above can also transformed to many other domain solutions.
For now, after we got an idea why we should replace Boolean for Date datatype we will focus about the details how we could reach this goal.
Dealing with standard SQL suggest that replacing a Database Management System (DBMS) for another one should not be a big issue. The reality is unfortunately a bit different. Not all available data types for date like Timestamp are really recommendable to use. By experience I prefer to use the simple java.util.Date to avoid future problems and other surprises. The stored format in the database table looks like: ‘YYYY-MM-dd HH:mm:ss.0’. Between the Date and Time is a single space and .0 indicates an offset. This offset describes the time zone. The Standard Central European Timezone CET has an offset of one hour. That means UTC+01:00 as international format. To define the offset separately I got good results by using java.util.TimeZone, which works perfectly together with Date.
Before we continue I will show you a little code snippet in Java for the OR Manager Hibernate and how you could create those table columns.
@Table(name = "ARTICLE")
public class ArticleDO {
@CreationTimestamp
@Column(name = "CREATED")
@Temporal(TemporalType.DATE)
private Date created;
@Column(name = "PUBLISHED")
@Temporal(TemporalType.DATE)
private Date published;
@Column(name = "DEFAULT_TIMEZONE")
private String defaultTimezone;
//Constructor
public ArticleDO() {
TimeZone.setDefault(Constraints.SYSTEM_DEFAULT_TIMEZONE);
this.defaultTimezone = "UTC+00:00";
this.published = new Date('0000-00-00 00:00:00.0');
}
public Date isPublished() {
return published;
}
public void setPublished(Date publicationDate) {
if(publicationDate != null) {
this.published = publicationDate;
} else {
this.published = new Date(System.currentTimeMillis());
}
}
}
//SQL
INSERT INTO ARTICLE (CREATED, PUBLISHED, DEFAULT_TIMEZONE)
VALUES ('1984-04-01 12:00:01.0', '0000-00-00 00:00:00,0', 'UTC+00:00);
Let get a bit closer about the listing above. As first we see the @CreationTimestamp Annotation. That means when the ArticleDO Object got created the variable created will initialized by the current time. This value never should changed, because an article can just once created but several times changed. The Timezone is stored in a String. In the Constructor you can see how the system Timezone could grabbed – but be careful this value should not trusted to much. If you have a user like me traveling a lot you will see in all the places I stay the same system time, because usually I never change that. As default Timezone I define the correct String for UTC-0. The same I do for the variable published. Date can also created by a String what we use to set our default zero value. The Setter for published has the option to define an future date or use the current time in the case the article will published immediately. At the end of the listing I demonstrate a simple SQL import for a single record.
But do not rush to fast. We also need to pay a bit attention how to deal with the UTC offset. Because I observed in huge systems several times problems which occurred because developer was used only default values.
The timezone in general is part of the internationalization concept. For managing the offset adjustments correctly we can decide between different strategies. Like in so many other cases there no clear right or wrong. Everything depends on the circumstances and necessities of your application. If a website is just national wide like for a small business and no time critical events are involved everything become very easy. In this case it will be unproblematic to manage the timezone settings automatically by the DBMS. But keep in mind in the world exist countries like Mexico with more than just one timezone. An international system where clients spread around the globe it could be useful to setup each single DBMS in the cluster to UTC-0 and manage the offset by the application and the connected clients.
Another issue we need to come over is the question how should initialize the date value of a single record by default? Because null values should avoided. A full explanation why returning null is not a good programming style is given by books like ‘Effective Java’ and ‘Clean Code’. Dealing with Null Pointer Exceptions is something I don’t really need. An best practice which well works for me is an default date – time value by ‘0000-00-00 00:00:00.0’. Like this I’m avoiding unwanted publishing’s and the meaning is very clear – for everybody.
As you can see there are good reasons why Boolean data types should replaced by Date. In this little article I demonstrated how easy you can deal with Date and timezone in Java and Hibernate. It should also not be a big thing to convert this example to other programming languages and Frameworks. If you have an own solution feel free to leave a comment and share this article with your colleagues and friends.
Since a long time the Java Script Object Notation [1] become as a lightweight standard to replace XML for information exchange between heterogeneous systems. Both technologies XML and JSON closed those gap to return simple and complex data of a remote method invocation (RMI), when different programming languages got involved. Each of those technologies has its own benefits and disadvantages. A good designed XML document is human readable but needs in comparing to JSON more payload when it send through the network. For almost every programming languages existing plenty implementations to deal with XML and also JSON. We don’t need to reinvent the wheel, to implement our own solution for handling JSON objects. But choosing the right library is not that easy might it seems.
The most popular library for JSON in Java projects is the one I already mentioned: Jackson [2]. because of its huge functionality. Another important point for choosing Jackson instead of other libraries is it’s also used by the Jersey REST Framework [3]. Before we start now our journey with the Java Frameworks Jersey and Jackson, I like to share some thoughts about things, I often observe in huge projects during my professional life. Because of this reason I always proclaim: don’t mix up different implementation libraries for the same technology. The reason is it’s a huge quality and security concern.
The general purpose for using JSON in RESTful applications is to transmit data between a server and a client via HTTP. To achieve that, we need to solve two challenges. First, on the server side, we need create form a Java object a valid JSON representation which we can send to the client. This process we call serialization. On the client side, we do the second step, which is exactly the opposite, we did on the server. De-serialization we call it, when we create a valid object from a JSON String.
In this article we will use on the server side and also on the client side Java as programming language, to deal with JSON objects. But keep in mind REST allows you to have different programming languages on the server and for the client. Java is always a good choice to implement your business logic on the server. The client side often is made with JavaScript. Also PHP, .NET and other programming Languages are possible.
In the next step we will have a look at the project architecture. All artifacts are organized by one Apache Maven Multi-Module project. It’s a good recommendation to follow this structure in your own projects too. The three artifacts we create are: api, server and client.
API: contain shared objects which will needed on the server and also client side, like domain objects and interfaces.
Server: producer of a RESTful service, depends on API.
Client: consumer of the RESTful service, depends on API.
Inside of this artifacts an layer architecture is applied. This means the access to objects from a layer is only allowed to the direction of the underlying layers. In short: from top to down. The layer structure are organized by packages. Not every artifact contains every layer, only the ones which are implemented. The following picture draws an better understanding for the whole architecture is used.
The first piece of code, I’d like to show are the JSON dependencies we will need in the notation for Maven projects.
In respect to the size of this article, I only focus how the JSON object is used in RESTful applications. It’s not a full workshop about RESTful (Micro) Services. As code base we reuse my open source GitHub project TP-ACL [4], an access control list. For our example I decided to sliced apart the Role – Functionality from the whole code base.
For now we need as first an Java object which we can serialize to an JSON String. This Domain Object will be the Class RolesDO and is located in the layer domain inside the API module. The roles object contains a name, a description and a flag that indicates if a role is allowed to delete.
@Entity
@Table(name = "ROLES")
public class RolesDO implements Serializable {
private static final long serialVersionUID = 50L;
@Id
@Column(name = "NAME")
private String name;
@Column(name = "DESCRIPTION")
private String description;
@Column(name = "DELETEABLE")
private boolean deleteable;
public RolesDO() {
this.deleteable = true;
}
public RolesDO(final String name) {
this.name = name;
this.deleteable = true;
}
//Getter & Setter
}
Listing 2
So far so good. As next step we will need to serialize the RolesDO in the server module as a JSON String. This step we will do in the RolesHbmDAO which is stored in the implementation layer within the Server module. The opposite direction, the de-serialization is also implemented in the same class. But slowly, not everything at once. lets have as first a look on the code.
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
public class RolesDAO {
public transient EntityManager mainEntityManagerFactory;
public String serializeAsJson(final RolesDO role)
throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
return mapper.writeValueAsString(role);
}
public RolesDO deserializeJsonAsObject(final String json, final RolesDO role)
throws JsonProcessingException, ClassNotFoundException {
ObjectMapper mapper = new ObjectMapper();
return (RolesDO) mapper.readValue(json, Class.forName(object.getCanonicalName()));
}
public List<RolesDO> deserializeJsonAsList(final String json)
throws JsonProcessingException, ClassNotFoundException {
ObjectMapper mapper = new ObjectMapper();
return mapper.readValue(json, new TypeReference<List>() {});
}
public List listProtectedRoles() {
CriteriaBuilder builder = mainEntityManagerFactory.getCriteriaBuilder();
CriteriaQuery query = builder.createQuery(RolesDO.class);
Root root = query.from(RolesDO.class);
query.where(builder.isNull(root.get("deactivated")));
query.orderBy(builder.asc(root.get("name")));
return mainEntityManagerFactory.createQuery(query).getResultList();
}
}
Listing 3
The implementation is not so difficult to understand, but may at this point could the first question appear. Why the de-serilization is in the server module and not in the client module? When the client sends a JSON to the server module, we need to transform this to an real Java object. Simple as that.
Usually the Data Access Object (DAO) Pattern contains all functionality for database operations. This CRUD (create, read, update and delete) functions, we will jump over. If you like to get to know more about how the DAO pattern is working, you could also check my project TP-CORE [4] at GitHub. Therefore we go ahead to the REST service implemented in the object RoleService. Here we just grep the function fetchRole().
The big secret here we have in the line where we stick the things together. As first the RolesDO is created and in the next line the DAO calls the serializeAsJson() Method with the RoleDO as parameter. The result will be a JSON representation of the RoleDO. If the role exist and no exceptions occur, then the service is ready for consuming. In the case of any problem the service send a HTTP error code instead of the JSON.
Complex Services which combine single services to a process take place in the orchestration layer. At this point we can switch to the client module to learn how the JSON String got transformed back to a Java domain object. In the client we don’t have RolesHbmDAO to use the deserializeJsonAsObject() method. And of course we also don’t want to create duplicate code. This forbids us to copy paste the function into the client module.
As pendant to the fetchRole() on the server side, we use for the client getRole(). The purpose of both implementations is identical. The different naming helps to avoid confusions.
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
public class Role {
private final String API_PATH
= "/acl/" + Constraints.REST_API_VERSION + "/role";
private WebTarget target;
public RolesDO getRole(String role) throws JsonProcessingException {
Response response = target
.path(API_PATH).path(role)
.request()
.accept(MediaType.APPLICATION_JSON)
.get(Response.class);
LOGGER.log("(get) HTTP STATUS CODE: " + response.getStatus(), LogLevel.INFO);
ObjectMapper mapper = new ObjectMapper();
return mapper.readValue(response.readEntity(String.class), RolesDO.class);
}
}
Listing 5
As conclusion we have now seen the serialization and de-serialisation by using the Jackson library of JSON objects is not that difficult. In the most of the cases we just need three methods:
serialize a Java object to a JSON String
create a Java object from a JSON String
de-serialize a list of objects inside a JSON String to a Java object collection
This three methods I already introduced in Listing 2 for the DAO. To prevent duplicate code we should separte those functionality in an own Java Class. This is known as the design pattern Wrapper [5] also known as Adapter. For reaching the best flexibility I implemented the JacksonJsonTools from TP-CORE as Generic.
package org.europa.together.application;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.core..JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;
public class JacksonJsonTools {
private static final long serialVersionUID = 15L;
public String serializeAsJsonObject(final T object)
throws JsonProcessingException {
try {
ObjectMapper mapper = new ObjectMapper();
return mapper.writeValueAsString(object);
} catch (JsonProcessingException ex) {
System.err.println(ex.getOriginalMessage());
}
}
public T deserializeJsonAsObject(final String json, final Class object)
throws JsonProcessingException, ClassNotFoundException {
try {
Class<?> clazz = Class.forName(object.getCanonicalName());
ObjectMapper mapper = new ObjectMapper();
return (T) mapper.readValue(json, clazz);
} catch (JsonProcessingException ex) {
System.err.println(ex.getOriginalMessage());
}
}
public List deserializeJsonAsList(final String json)
throws JsonProcessingException, ClassNotFoundException {
try {
ObjectMapper mapper = new ObjectMapper();
return mapper.readValue(json, new TypeReference<List>() {
});
} catch (com.fasterxml.jackson.core.JsonProcessingException ex) {
System.err.println(ex.getOriginalMessage());
}
}
}
Listing 6
This and much more useful Implementations with a very stable API you find in my project TP-CORE for free usage.
Sämtlich in einem Unternehmen aufgestellten Regeln und durchgeführten Aktivitäten stellen Prozesse dar. Deswegen kann auch pauschal gesagt werden, das die Summe der Prozesse eine Organisation beschreibt. Leider sind manchmal die Prozesse so kompliziert gestaltet, das diese sich negativ auf das Unternehmen auswirken. Was kann also getan werden um die Situation zu verbessern?
(c) 2022 Marco Schulz, JAVA aktuell Ausgabe 6
Laut ISO 900 Definition ist ein Prozess, ein Satz von in Wechselbeziehung stehenden Tätigkeiten. der Eingaben in Ergebnisse umwandelt. Dabei spielt es keine Rolle, ob der Prozess atomar ist, also nicht weiter zerlegt werden kann oder aus mehreren Prozessen zusammengesetzt wurde. An dieser Stelle ist es wichtig auch kurz auf einige Begriffe einzugehen.
Choreographie: beschreibt einzelne Operationen, aber nicht die Nachrichtenreihenfolge (Ablauf). Es behandelt die etablierte Kommunikation zwischen zwei Teilnehmern.
Orchestration: beschreibt die Reihenfolge und Bedingungen der aufrufenden Teilprozesse.
Konversation: beschreibt die Abfolgen zwischen Prozessen. Es wird die gesamte zulässige Kommunikation (Vollständigkeit) zwischen zwei Teilnehmern beschrieben.
Die aufgeführten Begrifflichkeiten spielen für die Beschreibung von Prozessen eine wichtige Rolle. Wenn sie beispielsweise die Idee haben die für Ihr Unternehmen wichtigen Geschäftsprozesse in einem Prozessbrowser visualisiert darzustellen, müssen Sie sich bereits im Vorfeld über die Detailtiefe der bereitgestellten Informationen im Klaren sein. Sollten Sie die Absicht hegen möglichst alle Informationen in so einem Schaubild einzubringen, werden Sie schnell feststellen wie sehr die Übersichtlichkeit darunter leidet. Wählen Sie daher immer für die benötigte Anwendung die geeignete Darstellung aus.
Ansichtssachen
Hier kommen wir auch schon zur nächsten Fragestellung. Was sind geeignete Mittel um Prozesse verständlich darzustellen. Aus persönlicher Erfahrung hat sich in meinen Projekten ein Darstellung über den Informationsfluss gut bewährt. Dazu wiederum nutze ich die Business Process Model Notation, kurz BPMN die für solche Zwecke geschaffen wurde. Ein frei verfügbares Werkzeug um BPMN Prozesse aufzuzeichnen ist der BigAzi Modeler [1]. Die Möglichkeit aus BPMN Diagrammen wiederum softwaregestützte Programme mittels serviceorientierter Architekturen (SOA) zu erzeugen ist für ein Großteil der Unternehmen weniger nutzbringend und nicht so einfach umzusetzen wie es auf den ersten Blick scheint. Viel wichtiger bei einer Umsetzung zur grafischen Darstellung interner Unternehmensprozesse sind die so zu tage geförderten versteckten Erkenntnisse über mögliche Verbesserungen.
Besonders Unternehmen, die eine eigenständige Softwareentwicklung betreiben und die dort angewendeten Vorgehensweisen, möglichst in einem hohen Grade automatisieren wollen, können den Schritt zur Visualisierung interner Strukturen selten auslassen. Die hier viel zitierten Stichwörter Continuous Integration, Continuous Delivery und DevOps haben eine sehr hohe Automatisierungsstufe zum Ziel. Um in diesem Bereich erfolgreiche Ergebnisse erreichen zu können, ist es unumgänglich möglichst einfache und standardisierte Prozesse etabliert zu haben. Das beschreibt auch das Paradoxon der Automatisierung.
Prozessautomation reduziert das Risiko, dass Fehler gemacht werden. Aber hochkomplexe Prozesse sind naturgemäß nur sehr schwer zu automatisieren!
Wenn Sie den Entschluss gefasst haben die hauseigenen Geschäftsprozesse zu optimieren benötigen Sie selbstredend zuerst eine realistische Analyse des aktuellen IST – Zustands um daraus den gewünschten SOLL – Zustand zu beschreiben. Sobald diese beiden wichtigen Punkte feststehen können Sie geeignete Maßnahmen ergreifen, mit der sie die Transformation vollziehen können.
Abbildung 1: Die Transformation von der Ausgangssituation hin zu Zielstellung.
Es wäre an dieser Stelle nicht sehr hilfreich verschiedene Vorgehnsmodelle zu beschreiben, wie eine solche Transformation von statten gehen kann. Solche Vorhaben sind stets sehr individuell und den tatsächlichen Gegebenheiten im Unternehmen geschuldet. Hier sei Ihnen nur ein wichtiger Ratschlag mit auf den Weg gegeben. Gehen Sie kleine einfache Schritte und vermeiden Sie es möglichst alles auf einmal umsetzen zu wollen. Manchmal entdecken Sie während einer Umstellung wichtige Details die angepasst werden müssen. Das gelingt Ihnen gefahrlos wenn Sie genügend Reserven eingeplant haben. Sie sehen auch hier spiegeln sich agile Gedanken wieder, die Ihnen die Möglichkeit geben direkt auf Veränderungen einzugehen.
Richten Sie Ihr Augenmerk vor allem auf den zu erreichenden Sollzustand. Im Großen und Ganzen wird zwischen zwei Prozesstypen unterschieden. Autonome Prozesse laufen im Idealfall vollständig automatisiert ab und erfordern keinerlei manuelles Eingreifen. Dem gegenüber stehen die interaktiven Prozess, welche an ein oder mehreren Stellen auf eine manuelle Eingabe warten um weiter ausgeführt werden zu können. Ein sehr oft angestrebtes Ziel für den SOLL – Zustand der Prozesslandschaften sind möglichst kompakte und robuste autonome Prozesse um den Automatisierungsgrad zu verbessern. Folgende Punkte helfen Ihnen dabei das gesteckte Ziel zu erreichen:
Definieren Sie möglichst atomare Prozesse, die ausschließlich einen einzigen Vorgang oder einen Teilaspekt eines Vorgangs beschreiben.
Halten Sie die Prozessbeschreibung möglichst sehr einfach und orientieren Sie sich dabei an vorhanden Standards und suchen Sie nicht nach eigenen individuellen Lösungen.
Vermeiden Sie so gut es möglichst jegliche manuelle Interaktion.
Wägen Sie bei Ausnahmen sehr kritisch ab, wie oft diese tatsächlich auftreten und suchen Sie nach möglichen Lösungen diese Ausnahmen mit dem Standartvorgehen abarbeiten zu können.
Setzen Sie komplexe Prozessmodelle ausschließlich aus bereits vollständig beschriebenen atomaren Teilprozessen zusammen.
Sicher stellen Sie sich die Frage, was es mit meinem Hinweis auf die Verwendung von etablierten Standards auf sich hat. Viele der in einem Unternehmen auftretenden Probleme wurden meist bereist umfangreich und bewährt gelöst. Nicht nur aus Zeit und Kostengründen sollte bei der Verfügbarkeit bereits etablierter Vorgehensmodelle kein eigenes Süppchen gekocht werden. So erschweren Sie zum einem den Wissenstransfer zwischen Ihren Mitarbeitern und zum anderen erschweren Sie die Verwendung von standardisierter Branchensoftware. Hierzu möchte ich Ihnen ein kleines Beispiel aus meinem Alltag vorstellen, wo es darum geht in Unternehmen möglichst automatisierte DevOps Prozesse für die Softwareentwicklung und den Anwendungsbetrieb zu etablieren.
Die Kunst des Loslassen
Die größte Hürde die ein Unternehmen hier nehmen muss, ist eine Neuorientierung an dem Begriff Release und dem dahinterliegenden Prozess, der meist eigenwillig interpretiert wurde. Die Abweichung von bekannten Standards hat wiederum mehrere spürbare Folgen. Neben erhöhtem Personalaufwand für die administrativen Eingriffe im Releaseprozess besteht auch stets die Gefahr durch unglückliche äußere Umstände in zeitlichen Verzug zum aktuellen Plan zu geraten. Ohne auf die vielen ermüden technischen Details einzugehen liegt das gravierendste Missverständnis in dem Glauben es gäbe nach dem Erstellen eines Releases noch die Möglichkeit die in der Testphase erkannten Fehler im selben Release zu beheben. Das sieht dann folgendermaßen aus: nach einem Sprint wird beispielsweise das Release 2.3.0 erstellt, welches dann ausgiebig in der Testphase auf Herz und Nieren überprüft wird. Stellt man nun ein Fehler fest, ist es nicht möglich eine korrigiert Version 2.3.0 zu erzeugen. Die Korrektur hat ein neues Release zur Folge, welches dann die Versionsnummer 2.3.1 trägt. Ein wichtiger Standard der hier zum Tragen kommt ist die Verwendung des Semantic Versioning, welcher jedem einzelnen Segment der Versionsnummer eine Bedeutung zuordnet. In dem hier verwendeten Beispiel zeigt die letzte Stelle die für ein Release durchgeführten Korrekturen an. Falls Sie sich etwas intensiver mit dem Thema Semantic Versioning beschäftigen mögen, empfehle ich dazu die zugehörige Internetseite [2].
Was aber spricht nun dagegen ein bereits geplantes und auf den Weg gebrachtes Release bei der Detektion von Fehlern nicht zu stoppen, zu korrigieren und ‘repariert’ erneut unter der bereits vergebenen Versionsnummer auf den Weg zu schicken? Die Antwort ist recht einfach. Der erhebliche Arbeitsaufwand, welcher ausschließlich manuell durchgeführt werden muss, um den Fehler wieder auszubügeln. Abgesehen davon wird Ihre gesamte Entwicklungsarbeit für das Folgerelease erheblich ausgebremst. Ressourcen können nicht frei gegeben werden und der Fortschritt beginnt zu stagnieren.
Deswegen ist es wichtig sich so zu disziplinieren, das ein bereits auf den Weg gebrachtes Release sämtliche Prozeduren durchläuft und erst im letzten Schritt dann die manuell ausgeführte Entscheidung getroffen wird, ob das Release für den Produktive Einsatz auch geeignet ist. Deswegen rate ich grundsätzlich dazu den Begriff Release Kandidat aus dem Sprachgebrauch zu streichen und besser von einem Production Kandidat zu sprechen. Diese Bezeichnung spiegelt den Releaseprozess viel deutlicher wieder.
Sollten sich währen der Testphase Mängel aufzeigen, gilt zu erst zu entscheiden wie schwerwiegend diese sind und deren Behebung ist zu priorisieren. Das kann soweit gehen, das direkt ein Korrekturrelease auf den Weg gebracht werden muss, während parallel der nächste Sprint abgearbeitet wird. Weniger gravierende Fehler können dann auf die nächsten Folgesprits verteilt werden. Wie das alles in der täglichen Praxis umgesetzt werden kann – habe ich letztes Jahr in meinem Vortag “Rolling Stones: Vom Release überrollt” auf der JCON präsentiert. Den Videomitschnitt finden Sie frei zugänglich im Internet [3].
Unter dem Gesichtspunkt der Prozessoptimierung bedeute es für das aufgeführte Beispiel des Release Prozesses, das der Prozess beendet wurde, wenn aus dem Sourcecode erfolgreich eine binäres Artefakt mit einer noch nicht belegten Versionsnummer erstellt werden konnte. Das so entstandene Release wird umgehend an einer zentralen Stelle veröffentlicht (deliverd), wo es in den Testprozess übergeben werden kann. Erst wenn der Testprozess mit dem Ergebnis abgeschlossen wurde, dass das erzeugte Release auch in Produktion verwendet werden darf erfolgt die Übergabe in den Deployment Prozess. Sie sehen, das was vielerorts als ein gesamter Prozess angesehen wird ist genau betrachtet eine Orchestration aus mindestens 3 eiegnständigen Prozessen.
Abbildung 2 : Continuous Delivery und Continuous Deployment.
Ein wichtiger Punkt den Sie In Abbildung 2 zum Thema DevOps ebenfalls herauslesen können ist, das der Schritt zwischen Continuous Delivery und Continuous Deployment besser nicht vollautomatisiert werden sollte, denn Deplyoment meint in diesem Kontext nicht das automatisierte bereitstellen der Anwendung auf allen verfügbaren Testinstanzen. Continuous Deployment meint in erste Linie ein automatisiertes Einsetzen der Anwendung in Produktion. Ob das immer eine gute Idee ist sollt sehr sorgfältig abgewogen werden.
Ein wertvoller Aspekt der Prozessbeschreibung in Organisationen ist die Ausarbeitung wichtiger Kriterien die erfüllt sein müssen, damit ein Prozess autonom ablaufen kann. Mit diesem Wissen können Sie bei der Evaluierung benötigter Werkzeuge sehr leicht einen Anforderungskatalog mit priorisierten Punkten erstellen, der einfach abgearbeitet wird. Kann das ins Auge gefasste Tool die aufgelisteten Punkte zufriedenstellend lösen und der aufgerufene Preis passt auch ins Budget, ist Ihre Suche erfolgreich beendet.
Fazit
Sehr oft wird mir entgegengebracht, das durch moderne DevOps Strategien der klassische Release Prozess obsolet geworden ist. Dem kann ich nicht zustimmen. Es mag wenige Ausnahmen geben, in den Unternehmen tatsächlich jede Codeänderung sofort in Produktion bringen. Aus Gründen der Gewährleistung und Haftung, kommt für viel Firmen ein so vollständig automatisiertes Vorgehen aber nicht in Frage. Auch der Datenschutz sorgt dafür, das die Bereich Entwicklung und Betrieb voneinander getrennt werden. Zudem benötigen umfangreiche Softwareprojekte auch eine strategische Planungsinstanz über die umzusetzenden Funktionalitäten. Diese Entscheidbarkeit wird auch künftig nicht beim Entwickler liegen, ganz gleich wie hervorgehoben der Punkt DevOps in der Stellenbeschreibung auch sein mag.
Wie Sie sehen ist das Thema der Prozessbeschreibung und Prozessoptimierung nicht ausschließlich ein Thema für produzierende Branchen. Auch der vielrorts detailreich beschriebene Softwareentwicklungsprozess hält einiges an Verbesserungspotenzial bereit. Ich hoffe ich konnte Sie mit meinen Zeilen ein wenig für das Thema sensibilisieren, ohne zu sehr ins technische verfallen zu sein.
Abstract: In the last decades, many standards were established to increase productivity during Software Lifecycle Management. All these techniques and methodologies promise a higher success rate in software projects which could affirm themselves in the case the involved protagonists are willing to follow the instances recommended. Semantic Versioning, for example, addresses the information leak between functional changes, BugFixes and compatibility of existing and future releases of artifacts. Diving deeper into the daily craftsmanship of software projects enables us to identify the Source Control Management Systems (SCM) as a big treasure box. Much information can be extracted from these repositories, which are currently ignored for project analyzing. Expressions on SCM Commit Messages represent a new formalism that is both human-readable and machine-processable. Such a standard also forms a bridge between the code base and the requirements management and release management, since these activities are identified by a freely expandable vocabulary in the SCM. Another advantage of this strategy is the clear and compact expressiveness for development teams. A very practical aspect of my proposal is the easy applicability of the presented solution in real software development projects. As with the Semantic Versioning methodology already mentioned, there are no additional technical requirements to be met, since commit messages are a fundamental function of SCM systems. This paper discuss the option to improve data collection for controlling software projects and knowledge sharing in collaborative teams.
To cite this article: Marco Schulz. Expressions for Source Control Management Systems. American Journal of Software Engineering and Applications. Vol. 11, No. 2, 2022, pp. 22-30. doi: 10.11648/j.ajsea.20221102.11
Download the PDF: https://www.sciencepg.com/journal/paperinfo?journalid=137&doi=10.11648/j.ajsea.20221102.11
1. Introduction
Thinking about SCM systems we have to keep in mind, that since the first roll out of CVS in the early 1990‘s and today, many things have changed. Searching the free online encyclopedia Wikipedia, presents a page ”Comparison of Version Control Software” which contains an overview of version control software of more than 30 SCM tools. This gives an idea why software companies usually have around three or more different SCM systems in work – of course the real amount depends on how many years they are in business.
The possibility to attach every revision in SCM Systems with a commit message allows the developer to inform other users with a short explanation of his work. This feature is extremely helpful by browsing the history manually in search of special code changes. If these commit messages well structured there exist a possibility to grab automated information of project growth. In this paper on expressions is introduced as solution for structured commit messages which could processed by software and also helps developers to resume their work more efficient.
The list of research on SCM is quite overwhelming and covers multiple aspects. The work of Walter F. Tichy on RCS [2] presents a deep fundamental insight into technical aspects of SCM systems. Abdullah Uz Tansel et al. gives in his research a brief history and builds a bridge to nowadays SCM systems [11]. The paper of Christian Bird et al. describes the ideas why companies deal with various SCM solutions [12]. Many existing papers like the one from Filip Van Rysselberghe and Serge Demeyer already identified SCM repositories as a significant information storage [5], which contains more than a simple history of source code. The approach from Louis Glassy to observe the growth of students in the software development process by using SCM techniques [6] demonstrates another method to grab implicit information from SCM. Alongside the fundamental research in software engineering, there exists a great resource of Blogs, articles and books from people who are directly involved in the topic. They describe experiences and best practice to make the next release come true, as referred towards the web resources in the footnotes. A small selection of related practitioners books is also included in the reference list.
Let us take a closer look at how processes for SCM could be improved. For this reason, section II defines the terminology of this paper and talks in detail about merging and branching strategies. Section III remind some basic knowledge on SCM and gives a simple idea about how complex build and deploy pipelines interact. Following this quick journey, section IV draws a picture about real problems that occur in software development projects and explains possible Points of Interest (POI) inside an SCM repository. These fundamentals allow a definition of the vocabulary we introduce in section V. A real world example will demonstrate in VI the cardinality of the expression and gives ideas about its usage. After all, section VII will reflect and summarize these thoughts. The last section talks about ideas how future work could be continued.
Figure 1: Branch and Merge.
2. Definitions
The definitions in this section are based on the English dictionary Merriam Webster with a contextual relation to SCM systems. The term Source Control Management System (SCM) is applied in this paper to describe tools like CVS, Subversion (SVN) or Git. Many other names have appeared over the years in literature for this type of tools. All these terms like Version Control System (VCS) or Revision Control System (RCS) are considered as equal to each other.
Artifact “A USUALLY SIMPLE OBJECT (SUCH AS A TOOL OR ORNAMENT) SHOWING HUMAN WORKMANSHIP OR MODIFICATION AS DISTINGUISHED FROM A NATURAL. OBJECT; “ESPECIALLY: AN OBJECT REMAINING FROM A PARTICULAR PERIOD”. In the context of SCM, an artifact is a binary result of the build process. Artifacts can be libraries, applications and so on.
Repository “A PLACE, ROOM, OR CONTAINER WHERE IS DEPOSITED OR STORED”. In software engineering a repository denotes a managed storage. We can distinguish repositories for source code and for binary artifacts.
Revision “A CHANGE OR A SET OF CHANGES THAT CORRECTS OR IMPROVES SOMETHING”. Each successful commit from a user to the SCM represents a change of the internal state in the SCM. These different states are revisions. Subversion for example increments an internal number after each commit [18]. This unique identifier is called revision number. Git on the other hand manages the revision number smarter and creates SHA-1 Hashes from each commit as an identifier [15]. This brings more flexibility for dealing with branches.
Release “TO GIVE PERMISSION FOR PUBLICATION, PERFORMANCE, EXHIBITION, OR SALE OF; ALSO: TO MAKE AVAILABLE TO THE PUBLIC”. A release defines a set of functional assertions for an artifact. When all functions are implemented, a test procedure is started to exclude as many failures as possible. After the termination of testing and corrections, the artifact gets packed for delivery. To distinguish the different versions of an artifact, it gets labeled by a unique version number. By convention, it is not allowed to have more than one artifact with the same version number.
Tag “A DESCRIPTIVE OR IDENTIFYING EPITHET”. -A Tag is a label to a special revision, like a release, and is used as bookmark.
Trunk “THE CENTRAL PART OF ANYTHING”. A trunk is a common convention and means the main branch, where the current development happens [17]. In Git this branch is called master for the local repository and orgin in the remote repository. Branching and Merging is one of the major feature in SCM systems and also a high sophisticated operation. It is not so unusual that developers and also Configuration Managers struggle with this. The paper of Shaun Phillips et al. contains a developer comment about the dealing with SCM and the pain of merging [10].
“We are a team of four senior developers (by which I mean we’re all over 40 with 20+ years each of development experience) and not one of us has had a positive experience in the past with branching the mainline… The branch is easy – it’s the merge at the end that’s painful.”
This shows that even persons with many years of experience need a detailed explanation of a seemingly trivial procedure. A simple understanding how branches typically have to be used and how they represent the evolution of a real software project is of high relevance for this paper. Figure 1 explains the optimal interaction between branches and the trunk which is described by Chuck Walrad and Darrel Strom as Branch by Release Model [3]. In addition to the context of branching and merging there is a version tree sample graph explained by Yongchang Ren et al. in their paper [8].
In order to give a comprehensive explanation of the process we assume a simple Java library project. As build tool Apache Maven is chosen which is successfully used for years by many different commercial and Open Source projects. Maven defines many standards for the software development process and implements them. Its success feature is a highly efficient dependency management.
The information about the artifact version number is managed in the pom.xml, the Maven build file. For this reason the POM has our special attention. In the context of Maven a versions number is labeled SNAPSHOT while it is still under development. This convention allows in collaborative teams the sharing of non official published artifacts. After removing the label SNAPSHOT the artifact is released. By convention it is not possible to have more than one artifact with the same version number. In section III this topic is discussed in more detail. For the moment it is necessary to know that this convention takes effect in collaborative processes. The correct way to share artifacts is the usage of a Repository Manager. The most common Repository Manager is Sonatype Nexus OSS which is used for Maven Central [19] to deliver dependencies. Nexus will refuse the request if a developer tries to publish an already existing release of an artifact. With this infrastructure it is not necessary to transfer binary artifacts to the SCM. This tool chain is a simple example for a highly complex infrastructure to build and deliver software in large companies.
In figure 1 the development starts with version 1.0-SNAPSHOT. After the release of this version, the development of the next version 1.1-SNAPSHOT continues in trunk. The revision of the released version 1.0 gets branched to fix some bugs. The branch will not be created automatically during the release, rather it gets created when there is a need, for example BugFixes. The branch will be named by its minor version 1.0 to stay flexible for further corrections. After a correct BugFix the changes get merged back to trunk and so on. It is very important to keep in mind, that after a release, no new functionality can be added to the versions 1.0.X, only corrections are allowed.
The merging of failure corrections can lead to complications if there already exist deployed versions. When a bug is detected down to an existing version it will be necessary to fix all following versions and increment their version number as part of the correction. For example if there exist released versions 1.0.2, 1.1.1, 1.2.3 & 2.0.1. and the fix has been done in version 1.0.2 it will have to be renamed 1.0.3 for release. The merge direction is always from the lower to the higher version which means that the version numbers of all following involved artifacts have to be increased. By this it can be assured that only fixes will be exchanged and no functionality is moving form an higher to a lower version within the merging process.
In this model the case of parallel feature development is missing. This happens when a very complex functionality is planned and the implementation cannot be finished in one release cycle. This especially often occurs in agile projects with a short time line between releases. Feature Branches address this requirement as well. The process is a simple extension of the Branch by Release Model. The Feature Branch will be created from the trunk and will be named like the feature. To test compatibility this branch at least needs to be merged from the trunk after each release. A merge can also be performed if the trunk provides important new features – whenever necessary.
A very useful advanced usage of branches is the stash command, that comes as build-in with Git. Indeed this feature is not so common but simple and powerful. Imagine a developer is working on some implementation with the urgency of having to deliver a BugFix for another release. He needs to switch his workspace to this branch but the current work needs to be saved without a direct commit to the trunk. The solution is create a branch and check in the current work and hence switch the branch for the fix. After all is done he will have to switch to the stashed branch, finish the work and merge the result to the trunk. An often observed procedure for developers are simultaneous checkouts of different branches and just switching the IDE workspace. By experience in large companies, this is very time consuming and error prone. By the law of Murphy, the only needed branch is the one not present in a local checkout collection.
To get in touch with branch models more profoundly, the website of the Git SCM [20] presents different branching workflows. Also at [21] exists a very detailed explanation for Git branch and merge best practices.
3. Quick Survey on SCM Basics
As described, there exists a huge amount of Source Control Management solutions. Even just picking out the most popular systems, we are able to identify many differences in detail. These may be the reasons why some tools have become more popular than others. Naturally, all of these systems do the job and are based on common ideas. A very early and fundamental work on SCM systems done by Tichy gives a deep insight about the Theory on how an SCM should be constructed [2]. Today, based on the approach of how things are done, we can classify them. Directory and file based systems, like Microsoft Visual Source Safe, are part of the less effective group of SCM. In commercial environments this group has low relevance because quite often it causes inconsistencies of the repository. This leads us to the category of Client-Server solutions. Client-Server SCM systems have two manifestations: centralizedand distributed. SVN is the most famous representative for centralized solutions. In new projects the choice of the day will very often be Git, a very popular distributed SCM tool. In “Transition from Centralized to Decentralized Version Control Systems” the authors describes why decentralized SCM systems are favored by developers [12]. Interviews of developers have shown the benefits and risks of applicated SCM systems. They deliver a well elaborated explanation why distributed SCM has a higher learning curve. This finding is a important principle for dealing with SCM.
SCM systems are designed to handle plain text files, like those used for source code. After a file has undergone configuration management and had an initial transfer into the repository, the system stores only a delta of the changes for every new transaction. With this requirement the repository is more efficient and needs less disk storage. This implies binary files like office documents should not be stored in SCM repositories because the system cannot calculate a delta and will always store a complete new copy of the file, if it has been changed. A solution for dealing with binaries, like dependencies or third party libraries, are Repository Managers which were introduced in section II.
Figure 2: Changes in the POM, based on Semantic Versioning.
At this point some performance issues for SCM have to be taken in consideration. This is of outstanding importance, because it defines how a repository should be organized. Large projects with a code repository up to 1 GB take a long time for a checkout, even though there is only a small subset of files that are chosen. 20 minutes and more are very common. The reason for this effect is the size of the repository itself. When it contains a lot of files it takes more time to calculate the internal tree. The best solution for a high performance repository is: Only text files and just one independent project or module per repository.
In continuation surges question how files are represented in a SCM. As an example we remember the small Java library project with the Maven build logic. The build logic is represented as an XML file and contains the entry <version>. This entry defines the version number of the artifact and starts with an initialization of 1.0.0-SNAPSHOT. The procedure to increase the version number strictly follows the Semantic Versioning. Figure 2 visualizes several steps between two releases. For each revision a label describes the process and the version number show the value in the POM file. This graphic is an extension with a detailed view of figure 1.
In reality things are never like explained in theory. Initial assumption often create a big dilemma in automation processes when it comes to execution. It is very easy to claim, that in a repository, the entry for version in the POM for releases is unique. For example, it means that there should not exist two revisions with a released version 1.0. But where humans work, mistakes will happen. For this reason we have the option to create tags into the SCM. Every revision in the SCM which represents a deployed release, will be tagged with the correct version number. Deployed releases are defined by a successful transfer of the binary artifact into the Repository Manager for collaborative usage.
4. Scenarios on Real Problems
We should focus our activities on special points in respect to the evolution of software projects. It is not useful to pay attention on each single revision. Let us highlight the Points of Interest (POI) and why they are special. In real projects with collaborative teams, it is quite common that a developer breaks the current build. The good news are: when Continuous Integration (CI) is applied in the process, these kind of problems will be detected very quickly and can be solved at the instance of them appearing [16]. But how a developer is able to break a build? This occurs when the changes get committed into the repository and some files are not included in the commit. A repair can easily and fast be done by adding a new commit with the missing files needed. In this case it is very important to realize that only the one who delivered an incomplete package is able to add the missing parts. Problems arise when this happens on a Friday evening and the person responsible is leaving the office for vacations the next two or tree weeks without checking that everything is in order, causing unnecessary pain in the continuation of the project. These things happen much more often than anyone would expect.
Another effect is called fast shots. These small and often repeated commits typically change only a few lines in just one or two files. This happens when a user for some reason is not able to test his code or settings locally on his own machine. A simple scenario could be the manipulation of the CI Server build output without direct access.
A work flow for developers is the usage of particular commits in order to preserve intermediate steps of the work and allow an easy rollback. This procedure is only applicable in distributed systems or in environments without collaboration. The effect is quit similar. It will produce many revisions inside the SCM, which could get summarized to a single revision.
The Continuous Delivery approach for modern Web Applications is a quite different method compared to the classical release process [14]. This technique requires special strategies like the Feature Toggle Pattern [22] and a highly automated deploy pipeline. Also the usage of the SCM system is very advanced. Each feature is developed in its own branch and the Configuration- or Build Manager creates for each deployment a proper Integration Branch. The biggest challenge in this methodology are fast responses towards urgent problems arising. In the worst case it could be necessary to push out very quickly a new deployment with a full or partial rollback. During deployments database changes are very critical. This aspect could be discussed in a further paper. Databases are not implicitly part of the SCM, but there also exist techniques [23] to keep them under configuration management.
Figure 3: Structure of a commit naming.
As mentioned before, a release R inside an SCM is defined by several commits to the SCM. These commits are identified by the revision r. The lowest amount of revisions between two release is one, but there is no limit concerning to the upper boundary. Special Points of Interests inside an SCM are released revisions which can formally defined by (2).
R := {r 1, r 2, r 3, r n+1,…, r x } (1)
POI:= ∆ Release (R; R + 1) (2)
By this interpretation we are able to develop metrics which show a real project growth and do not just produce an output [13]. The paper of P. Kaur and H. Singh contains a collection of metrics related to their VVCT SCM [9]. An adapted suggestion for possibilities to compare project evolution is:
the amount of BugFix releases in a minor branch,
an count of revisions between two release,
the growth between minor and major release (e.g. Line of Codes),
a direct comparison between the current trunk and a previous release,
two selected releases,
a comparison of an release R and its replacement.
For example the amount of BugFix releases for a minor release allows a conclusion about the quality situation of a project. It is very important to understand the reasons to improve program stability and reduce the number of BugFixes. A classification for changes is described by Swanson [1]. An overview of the project based on these classifications of BugFixes should detect the issues that have to be changed to accomplish high quality.
5. A Vocabulary for SCM Commit Messages
In the early times SCM systems were used for synchronizing source code between developers. Typically users were not paying too much attention to write well formulated explanations about their changes. In many instances they were not leaving any description about what they did. Another extreme was that comments like update build logic frequently appeared in the history. An explanation of everything and nothing without saying what was changed or why. It could either be a version update of an existing library or the addition of a new dependency leading to a heavy time-consuming work in order to identify the points of interest in the commit history. Manual checks between the version with a Diff Tool would be necessary to locate the Line of Code that may have to be changed again. Guidelines have been introduced on how to write a well formulated commit message to solve this problems. A short selection of these guides published on the internet: [24, 25, 26] It was discovered by companies that the approach to apply well formulated descriptions of SCM revisions can improve productivity in teams. By exploring new projects on Source Code Hosting Services like GitHub or Sourceforge the quality of commit messages was increasing in the last years.
Based on these recommendations and the experience gained as of today, a vocabulary should be introduced for writing easier and more efficient commit messages. This simple-to-use standardization could help to visualize the evolution of a project more clearly. By very precise and short explanation of every revision readers do not get flooded with information. This allows analysts to see patterns of process leaks more quickly and increases the team productivity. The usage of a defined structure also allows an automatism to parse the commit messages. The result can generate programmatic presentations of diagrams readable by humans. Naturally this approach is not only limited to SCM. Another usage could be for communication in meetings with strict time limitations, for example in the agile method Scrum.
The vocabulary for SCM Commit Messages follows a defined structure which is shown in figure 3. The composition contains a mandatory first line and includes a FunctionID, label and a short specification. The second and third line is optional and contains the TaskID from the Issue Management System and a description of the more detailed explanation. Our suggestion for the vocabulary covers most SCM work flows. It may will be that some companies need adoptions to implement this solution in their processes. For this reason the definition is flexible and allows extensions.
#INIT – the repository or a release.
repro:documentation / configuration…
archetype:jar / war / ear / pom / zip…
version:<version>
#IMPLEMENT – a functionality.
function:<clazz>
#CHANGE – a functionality.
function:<clazz>
#EXTEND – a functionality.
function:<clazz>
attach:<clazz>
#BUGFIX – a functionality.
priority:critical / medium / low / design
#REVIEW – an implementation.
refactor:<function>
analyze:<quality>migrate:<function>
format:<source>
#RELEASE – an artifact.
version:<version>
#REVERT – a commit.
commit:<id>
#BRANCH – create.
create:<name>
stash:<branch>
#MERGE – from another branch.
from:<branch>
to:<branch>
#CLOSE – a branch.
branch:<name>
As first entry a FunctionID is recommended and not the TaskID of the Issue Management. This decision is based on the experience that functionality could spread in different tasks. In longtime projects it could happen that for some reason the Issue Management System needs to be replaced by another one. Not all projects are connected to Issue Management, especially when they are small or just a prototype. These circumstances proved to be decisive to define the TaskId as optional and move it to the second line. With a FunctionID it is easier to identify parts that should be linked. Sometimes there exist transfers into the repository that cannot be assigned to a dedicated function. These commits are often related to activities of the Build- and Configuration Manager. As best practice an ID should be established which corresponds to these activities. Some examples related to the defined labels are:
[CM-00] INIT;
[CM-10] REVIEW;
[CM-20] BRANCH;
[CM-30] MERGE;
[CM-40] RELEASE;
[CM-50] build management.
The mightiness of this approach is its simplicity and how it can be included in existing projects. The rule set does not contain any additional complexity and the process is quite easy to understand. A short example will demonstrate the usage and a full example is provided in section VI. A change in the POM file to update the version of the test framework could be commented as follows:
[CM-50] #CHANGE ’function:pom’
<QS-23231>
{Change version number of the dependency JUnit from 4 to 5.0.2}
6. Release Process
The sample project in section II is not only fictive. The Together Platform (TP) available on GitHub [26] was initiated to study techniques on real conditions. Hence Git is the SCM tool of the choice. As client SmartGit is recommended because of platform independence and it offers plentiful advanced functionality.
For better comprehension of our approach of writing commit expressions we use the TP-CORE project, from initialization of the repository to its first release. No TaskIDs for the revisions exist due to the project not being connected to an Issue Management System. We use an excerpt of TP-CORE to demonstrate the approach because between the initial commit and the first published release 1.0.2 exist over 70 revisions in the repository. The project also contains a set of 12 functions which do not need to be included completely in our sample. Only three functions were selected for demonstration:
CORE-01 Logger;
CORE-02 genericDAO;
CORE-05 ApplicationConfiguration.
This cuts the revisions in half and shows enough complexity avoiding readers falling asleep.
The condition for a first release was the implementation of all 12 functionalities. The overall test coverage has reached more than 85%. Code smells detected with checks by Findbugs, Checkstyle, PMD et cetera have been removed. For an facilitate explanation, we add a revision number before the FunctionID. TP-CORE Commit Messages:
01[CM-00] #INIT ’archtype:jar’
{Initial the repository for Java JAR library.}
02[CORE-01] #IMPLEMENT ’function:Logger’
{Application wide standard logger.}
03[CORE-02] #IMPLEMENT
{Generic Data Access Object Pattern for centralized database access.}
04[CORE-05] #IMPLEMENT ’function:AppConfigDO’
{Domain Object for application configuration.}
05[CM-10] #REVIEW ’analyze:quality’
{Formatting, fix Checkstyle hints, JavaDoc & test coverage}
06[CORE-05] #IMPLEMENT ’function:ConfigurationDAO’
{Add the ConfigurationDAO implementation.}
07[CORE-05] #EXTEND ’attach:tests’
{Create test cases for Bean Validation.}
08[CORE-01] #EXTEND ’function:Logger’
{Add new Method to detect the configured LogLevel.}
09[CORE-05] #EXTEND ’function:AppConfigDO’
{Change Primary Key to UUID and extend tests.}
10[CORE-05] #CHANGE ’function:AppConfigDO’
{Rename to ConfigurationDO and define table indexes.}
11[CORE-02] #EXTEND ’function:GenericDAO’
{Add flushTable, countEnties and optimize.}
12[CORE-05] #EXTEND ’attach:tests’
{Update test cases for application configuration.}
13[CORE-05] #EXTEND ’function:ConfigurationDAO’
{Update the implementation for ConfigurationDAOImpl.}
14[CORE-01] #EXTEND ’function:Logger’
{Add method for exception handling.}
15[CORE-05] #EXTEND ’function:ConfigurationDO’
{Add field mandatory.}
16[CM-10] #REVIEW ’migrate:JUnit’
{Migrate Test cases from JUnit4 to JUnit5.}
17[CM-10] #REVIEW ’analyze:quality’
{Fix JavaDoc, Checkstyle & Findbugs.}
18[CM-50] #EXTEND ’function:POM’
{Update SCM connection to GitHub.}
19[CM-50] #EXTEND ’attach:APIguards’
{Attach annotation for API documentation.}
20[CORE-05] #REVIEW ’refactor:ConfigurationDO’
{FindBugs: optimize constructor parameters.}
21[CORE-02] #BUGFIX ’priority:design’
{Fix FindBugs hint: visible modifier.}
22[CM-50] #EXTEND ’attach:site’
{Extend MVN site configuration.}
23[CORE-02] #BUGFIX ’priority:high’
{Fix spring DAO configuration.}
24[CORE-05] #IMPLEMENT ’function:ConfigurationService’
{Implement basic functionality for
ConfigurationService.}
25[CM-10] #REVIEW ’analyze:quality’
{Remove all compiler warnings, FindBugs,
Checkstyle & PMD Hits.}
26[CORE-05] #EXTEND
’attach:ConfigurationService’
{Add JGiven test scenarios.}
27[CM-40] #RELEASE ’version:1.0’
{Release artifact to version 1.0}
28[CM-40] #RELEASE ’version:1.0.1’
{Change POM GroupId to Maven Central conventions.}
29[CM-00] #INIT ’version:1.1’
{Start implementation of version 1.1.0.}
30[CM-50] #MERGE ’from:1.0.1’
{Integrate GAV POM changes to trunk.}
31[CM-40] #RELEASE ’version:1.0.2’
{Include PGP signing.}
32[CM-20] #CHANGE ’function:Constraints’
{Add Constraints.VERSION to 1.1}
33[CORE-01] #EXTEND ’function:Logger’
{Default loader for logback.xml configuration files in the application DIR.}
Considering the previous example, we see that a limitation to around 80 – 100 characters for the first line is recommendable. Displaying the history with any client could get very messy if the first line has no size restrictions. The log output of the commit messages does not display the branch and tag operation, a behavior of Git. These revisions do not appear in any history list by browsing GitHub. Revision 28 is a branch based on revision 27. The branch is named as 1.0. Releases are published in consonance with the convention to be labeled, revision 31 tagged as Release 1.0.2. The revisions 28 and 31 are part of branch 1.0.
In this constellation we are able to see an important detail for dealing with branches. A branch will only be created when it is necessary. Usually BugFix branches do not have their own build plans on CI Servers and are managed manually. The primary arguments for this practice are to reduce the administrative overhead for the CI Servers. Companies that orchestrate their applications by web services or modules loose capacities by binding their recourses in this kind of activities.
7. Conclusion
“There is nothing permanent except change.” – Heraclitus
The whole infrastructure of commercial software projects contains a lot of independent fragments which share information over all development cycle. In projects we are overloaded by documentation production processes. The high amount of all this information inhibits profoundly comprehension and handling capabilities. Applications are getting more complex and bigger resulting in the necessity to establish more efficient ways to deal with information accumulation. There exists a giant overhead of managing documents like release notes, release plan, issue management, quality reports, statistics & metrics, documentation, architectural documents and BugFix lists. Typically each tool stores its data in its own structure. This makes changes to other tools, that might fit better, risky and expensive.
Companies know the effect that developers feel uncomfortable having to track their work in Issue Management tools like JIRA resulting in them trying to hide their part of the work flow as much as possible. Tasks will be opened up when they are almost done or already finished. The information on how many project days were spent for a function covers more the expectations and less the reality with the intent that developers can escape a bit from the daily pressure of productivity. Often developers are forced to spend their time with data acquisition for management controlling instead of programming resulting in low cost efficiency of a project and even additional and unplanned costs. Developers dislike this kind of activities because it keeps them away from their actual work: development. This is what makes the simple approach towards human readable and machine processable commit messages attractive and more convenient. The most important fact is that no extra costs are generated applying this method to existing processes.
We are enabled to generate several reports based on real data if SCM repositories can be populated with additional information. Impact assessments could be more efficient and accurate when they are created by facts and not emotionally blended.
Future Work
The idea to make information inside SCM systems more transparent is not just limited to commit messages. Another obvious point for future research is the history command. In the paper of Abram Hindle and Daniel M. German a query language for source control is introduced [7]. The idea of SCM Language could be picked up and transformed applying it to a specific solution. This work would use the Domain Driven Development paradigm to model an own SCM language based on Domain Specific Language (DSL) concepts – leading to the discovery of real world DSL solutions allowing for quick construction of a viable prototype or application based upon certain specifications.
Also a point which boldly comes to mind after reading the paper of Fischer et al., is the inclusion of released information into SCM [4]. This approach should not fully be automated due to its requirement of an advanced knowledge about branching and merging. A small self written extension could be a probable solution. A short tutorial 17 for Git suggests certain possibilities.
Acknowledgements
Special thanks to Joachim Reiter and Harald Kaufmann for spending their time to review this document. Their feedback was very productive.
References
[1] E. Burton Swanson, 1978, The Dimension of Maintenance.
[2] Walter F. Tichy, 1985, RCS – A System for Version Control.
[3] Chuck Walrad and Darrel Strom, 2002, The Importance of Branching Models in SCM.
[4] Michael Fischer, Martin Pinzger, Harald Gall, 2003, Populating a Release History Database from Version Control and Bug Tracking Systems.
[5] Filip Van Rysselberghe and Serge Demeyer, 2004, Mining Version Control Systems for FACs (Frequently Applied Changes).
[6] Louis Glassy, 2005, Using version control to observe student software development processes.
[7] Abram Hindle and Daniel M. German, 2005, SCQL: a formal model and a query language for source control.
[8] Yongchang Ren, Tao Xing, Qiang Quan, Ying Zhao, 2010, Software Configuration Management of Version Control Study Based on Baseline.
[9] Parminder Kaur and Hardeep Singh, 2011, A Model for Versioning Control Mechanism in Component- Based Systems
[10] Shaun Phillips, Jonathan Sillito, Rob Walker, 2011, Branching and merging: an investigation into current version control practices.
[11] Abdullah Uz Tansel and Ali Koc, 2011, A Survey of Version Control Systems.
[12] Christian Bird et al., 2014, Transition from Centralized to Decentralized Version Control Systems A Case Study on Reasons, Barriers, and Outcomes.
[13] Norman E. Fenton and Shari Lawrence Pfieeger, 1997, PWS Publishing Company, Software Metrics – A Rigorous and Practical Approach 2nd Edition, ISBN O·534·95425·1.
[14] Jez Humble and David Farley, 2010, Addison-Wesley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, ISBN 0-321-60191-2.
[15] Scott Chacon and Ben Straub, 2014, Apress, Pro Git 2nd Edition, ISBN 978-1-4842-0077-3.
[16] Mike Clark, 2004, The Pragmatic Bookshelf, Pragmatic Project Automation, ISBN 0-9745140-3-9.
[17] Dave Thomas and Andy Hunt, 2003, The Pragmatic Bookshelf, Pragmatic Version Control with CVS, ISBN 0-9745140-0-4.
[18] Mike Mason, 2010, The Pragmatic Bookshelf, Pragmatic Guide to Subversion, ISBN 1-934356-61-1.
[19] https://search.maven.org
[20] https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows
[21] https://nvie.com/posts/a-successful-git-branching-model/
[22] https://www.martinfowler.com/articles/feature-toggles.html
[23] https://flywaydb.org
[24] https://chris.beams.io/posts/git-commit/
[25] http://who-t.blogspot.mx/2009/12/on-commit-messages.html
[26] https://github.com/ElmarDott/TP-CORE/
Biography
Marco Schulz, also kown by his online identity Elmar Dott is an independent consultant in the field of large Web Application, generally based on the JavaEE environment. His main working field is Build-, Configuration- & Release-Management as well as software architecture. In addition his interests cover the full software development process and the discovery of possibilities to automate them as much as possible. Over the time of the last ten years he has authored a variety of technical articles for different publishers and speaks on various software development conferences. He is also the author of the book “Continuous Integration with Jenkins” published 2021 by Rheinwerk.
When we have a look at OWASP’s top 10 vulnerabilities [1], SQL Injections are still in a popular position. In this short article, we discuss several options on how SQL Injections could be avoided.
When Applications have to deal with databases existing always high-security concerns, if an invader got the possibility to hijack the database layer of your application, he can choose between several options. Stolen the data of the stored users to flood them with spam is not the worst scenario that could happen. Even more problematic would be when stored payment information got abused. Another possibility of an SQL Injection Cyber attack is to get illegal access to restricted pay content and/or services. As we can see, there are many reasons why to care about (Web) Application security.
To find well-working preventions against SQL Injections, we need first to understand how an SQL Injection attack works and on which points we need to pay attention. In short: every user interaction that processes the input unfiltered in an SQL query is a possible target for an attack. The data input can be manipulated in a manner that the submitted SQL query contains a different logic than the original. Listing 1 will give you a good idea about what could be possible.
SELECT Username, Password, Role FROM User
WHERE Username = 'John Doe' AND Password = 'S3cr3t';
SELECT Username, Password, Role FROM Users
WHERE Username = 'John Doe'; --' AND Password='S3cr3t';
Listing 1: Simple SQL Injection
The first statement in Listing 1 shows the original query. If the Input for the variables Username and Password is not filtered, we have a lack of security. The second query injects for the variable Username a String with the username John Doe and extends with the characters ‘; –. This statement bypasses the AND branch and gives, in this case, access to the login. The ‘; sequence close the WHERE statement and with — all following characters got un-commented. Theoretically, it is possible to execute between both character sequences every valid SQL code.
Of course, my plan is not to spread around ideas that SQL commands could rise up the worst consequences for the victim. With this simple example, I assume the message is clear. We need to protect each UI input variable in our application against user manipulation. Even if they are not used directly for database queries. To detect those variables, it is always a good idea to validate all existing input forms. But modern applications have mostly more than just a few input forms. For this reason, I also mention keeping an eye on your REST endpoints. Often their parameters are also connected with SQL queries.
For this reason, Input validation, in general, should be part of the security concept. Annotations from the Bean Validation [2] specification are, for this purpose, very powerful. For example, @NotNull, as an Annotation for the data field in the domain object, ensure that the object only is able to persist if the variable is not empty. To use the Bean Validation Annotations in your Java project, you just need to include a small library.
Perhaps it could be necessary to validate more complex data structures. With Regular Expressions, you have another powerful tool in your hands. But be careful. It is not that easy to write correct working RegEx. Let’s have a look at a short example.
public static final String RGB_COLOR = "#[0-9a-fA-F]{3,3}([0-9a-fA-F]{3,3})?";
public boolean validate(String content, String regEx) {
boolean test;
if (content.matches(regEx)) {
test = true;
} else {
test = false;
}
return test;
}
validate('#000', RGB_COLOR);
Listing 3: Validation by Regular Expression in Java
The RegEx to detect the correct RGB color schema is quite simple. Valid inputs are #ffF or #000000. The Range for the characters is 0-9, and the Letters A to F. Case insensitive. When you develop your own RegEx, you always need to check very well existing boundaries. A good example is also the 24 hours time format. Typical mistakes are invalid entries like 23:60 or 24:00. The validate method compares the input string with the RegEx. If the pattern matches the input, the method will return true. If you want to get more ideas about validators in Java, you can also check my GitHub repository [3].
In resume, our first idea to secure user input against abuse is to filter out all problematic character sequences, like — and so on. Well, this intention of creating a blocking list is not that bad. But still have some limitations. At first, the complexity of the application increased because blocking single characters like –; and ‘ could causes sometimes unwanted side effects. Also, an application-wide default limitation of the characters could cost sometimes problems. Imagine there is a text area for a Blog system or something equal.
This means we need another powerful concept to filter the input in a manner our SQL query can not manipulate. To reach this goal, the SQL standard has a very great solution we can use. SQL Parameters are variables inside an SQL query that will be interpreted as content and not as a statement. This allows large texts to block some dangerous characters. Let’s have a look at how this will work on a PostgreSQL [4] database.
DECLARE user String;
SELECT * FROM login WHERE name = user;
Listing 4: Defining Parameters in PostgreSQL
In the case you are using the OR mapper Hibernate, there exists a more elegant way with the Java Persistence API (JPA).
Listing 5 is shown as a full example of Hibernate using JPA with the criteria API. The variable for the user input is declared in the first line. The comments in the listing explain the way how it works. As you can see, this is no rocket science. The solution has some other nice benefits besides improving web application security. At first, no plain SQL is used. This ensures that each database management system supported by Hibernate can be secured by this code.
May the usage looks a bit more complex than a simple query, but the benefit for your application is enormous. On the other hand, of course, there are some extra lines of code. But they are not that difficult to understand.
