Especially in medium-sized companies, big data is sometimes seen as a kind of "jack of all trades". The assessment "Today you just do it with BigData, Spark, SMACK stack, Hortonworks, in the cloud with AWS" often leads to very complex solutions with high hardware requirements. An effort that in many cases would not even be necessary or even results in unsuitable solutions. Especially as the costs often significantly exceed the benefits. Against this background classic technologies such as relational databases were replaced by new technologies, products and paradigms in the big data environment into the background.
Based on this, we will examine classic technologies that are classified in the literature as not efficient in the big data environment in terms of scaling options. The aim is to to validate how, for example, the benefits of big data tools can also be achieved with relational databases. and whether there are clear indicators as to when it actually makes sense to rely on big data technologies.
It doesn't always have to be big data
As a general rule, the introduction of big data technologies means that the IT infrastructure must also be adapted in order to be able to operate the applications optimally.
Such an adaptation to Big data technology is particularly necessary when it comes to analyzing semi-structured or unstructured data. Traditional technologies such as relational databases are less suitable here due to their technical restrictions, as they can only be used to work directly with structured data.
However, if structured data is already available, it does not necessarily have to be a big data technology. In this case, the use case should first be used to assess whether an optimization of the existing technology or an adaptation of the question is sufficient. A yes/no decision may be sufficient instead of an exact calculation with a fine-grained probability calculation.
3-V model as an indicator for big data
There is no clear classification of when big data technologies should be used. However, based on the so-called 3 V model possible to recognize, whether the use of big data technologies would be helpful.
(Source: https://gi.de/informatiklexikon/big-data/)
The metrics of data volume, speed and variety characterize the requirements for a use case. Depending on the characteristics of the three properties, it is possible to assess whether modern technologies should be used.
If the highest or second-highest requirements exist in several data areas (outermost and second ring from the outside), the use of modern technologies or big data makes sense. However, if only individual requirements in one area high requirements (for example "particularly fast, but no high data volume", or "very high data volume per batch job"), must be Each application is assessed individually whether big data technologies are suitable or whether it is better to optimize classic technologies.
Optimization options for classic technologies
Best practices for exploiting the scaling potential of relational databases:
- The data in relational systems is generally unsorted, which means that creating indices can lead to faster access times.
- Moving partial relations to the working memory can lead to faster data processing, as the data is no longer transferred between the working memory and the hard disk memory.
- The creation of partitions and replications leads to parallel or individual access to different sub-relations of a database. This ensures parallel processing or only relevant data of a relation is used for analysis.
- Setting up a master-slave architecture is based on distributed data management, which is also used by Hadoop. This allows data sets to be distributed on different servers and parallel analyses to be carried out by splitting the load of a query.
- Materialized views can be used for the "pre-processing" of the data and for the subsequent presentation of the results in order to save time.
Conclusion:
In many situations, big data technologies can offer a solution that would not have been possible without them. However, the decision to use such technologies must be carefully considered, as their use is costly and time-consuming. The 3-V model can help to decide whether leaner and therefore cheaper approaches can also lead to the same goal.
Co-author Bartu Uysal
Find out more about data integration here
These other blog posts may also be of interest:
IoT process optimization and big data (part 1)
