Effective business intelligence and high-quality data analysis. Business Information Analysis - Fundamentals Big Data Connectors

Small business in the CIS countries does not yet use data analysis for business development, determining correlations, searching for hidden patterns: entrepreneurs manage to get by with the reports of marketers and accountants. Small and semi-medium-sized business leaders rely more on their intuition than on analysis. But at the same time, analytics have huge potential: it helps to reduce costs and increase profits, make decisions faster and more objectively, optimize processes, better understand customers and improve the product.

An accountant is not a substitute for an analyst

Small business leaders often assume that the reports of marketers and accountants adequately reflect the activities of the company. But it is very difficult to make a decision on the basis of dry statistics, and an error in calculations without specialized education is inevitable.

Case 1. Post-analysis of promotional campaigns. For the New Year, the entrepreneur announced a promotion, within the framework of which certain goods were offered at a discount. After assessing the revenue for the New Year period, he saw the sales increase and was delighted with his resourcefulness. But let's take all the factors into account:

  • Sales grow especially strongly on Friday, the day when revenue is highest - this is a weekly trend.
  • Compared to the growth in sales that usually occurs under New Year, then the gain is not so great.
  • If we filter out promotional items, it turns out that the sales figures have deteriorated.

Case 2. Research of turnover. At the store women's clothing difficulties with logistics: the goods are in short supply in some warehouses, and in some they have been lying for months. How to determine, without analyzing sales, how many trousers to bring to one region, and how many coats to send to another, while getting the maximum profit? To do this, you need to calculate the turnover, the ratio of the speed of sales and the average inventory for a certain period. To put it simply, turnover is an indicator of how many days a store will take to sell a product, how quickly the average stock is sold, how quickly the product pays for itself. It is economically unprofitable to store large reserves, as it freezes capital and slows down development. If the stock is reduced, there may be a shortage and the company will again lose profit. Where can you find the golden mean, the ratio at which the product does not stagnate in the warehouse, and at the same time, you can give a certain guarantee that the customer will find the desired unit in the store? To do this, the analyst must help you determine:

  • desired turnover,
  • turnover dynamics.

When settling with suppliers with a deferral, it is also necessary to calculate the ratio of the credit line and turnover. Turnover in days = Average inventory * number of days / Turnover for this period.

Calculation of the remaining assortment and the total turnover by stores helps to understand where it is necessary to move a part of the product. It is also worth calculating what the turnover rate for each unit of the assortment is, in order to make a decision markdown with a reduced demand, additional order with an increased demand, moving to a different warehouse. By categories, you can develop a report on turnover in this form. It can be seen that T-shirts and jumpers are sold faster, but coats - for a long time. Will an ordinary accountant be able to do this kind of work? We doubt it. At the same time, the regular calculation of turnover and the application of the results can increase profits by 8-10%

In what areas is data analysis applicable?

  1. Sales. It is important to understand why sales are going well (or bad), what the dynamics are. To solve this problem, you need to research the factors that influence profit and revenue - for example, analyze the length of the check and the revenue per customer. Such factors can be investigated by product groups, seasons, stores. You can identify highs and sales pits by analyzing returns, cancellations, and other transactions.
  2. Finance. Monitoring indicators is necessary for any financier to monitor cash flow and allocate assets across various areas of business. This helps to assess the efficiency of taxation and other parameters.
  3. Marketing. Any marketing company needs forecasts and post-analysis of stocks. At the stage of developing an idea, you need to determine the groups of goods (control and target) for which we are creating an offer. This is also a job for a data analyst, since an ordinary marketer does not have the necessary tools and skills for good analysis. For example, if the total revenue and the number of buyers for the control group are the same in comparison with the target group, the promotion did not work. Interval analysis is needed to determine this.
  4. Control. Leadership is not enough for a company leader. In any case, quantitative assessments of the work of personnel are necessary for the competent management of the enterprise. It is important to understand the efficiency of payroll management, the ratio of salaries and sales, as well as the efficiency of processes - for example, the workload of cash registers or the employment of loaders during the day. This helps to properly manage working hours.
  5. Web analysis. The site needs to be properly promoted in order for it to become a sales channel, and this requires the right promotion strategy. This is where web analysis comes in. How to use it? Study the behavior, age, gender and other characteristics of customers, activity on certain pages, clicks, traffic channel, the effectiveness of mailings, etc. This will help improve your business and website.
  6. Assortment management. ABC analysis is essential for assortment management. The analyst must distribute the product according to its characteristics in order to conduct this type of analysis and understand which product is the most profitable, which is the basis, and which one is worth getting rid of. To understand the stability of sales, it is good to conduct an XYZ analysis.
  7. Logistics. A better understanding of procurement, goods, their storage and availability will give the study of logistics indicators. Losses and needs of goods, inventory is also important to understand for successful business management.

These examples show how powerful data analysis can be, even for small businesses. An experienced CEO will increase the company's bottom line and benefit from the smallest insights by using data analytics correctly, and the manager's job will be greatly simplified by visual reports.

The main goal of any data analysis is to find and discover patterns in the volume of data. In business analysis, this goal becomes even broader. It is important for any leader not only to identify patterns, but also to find their cause. Knowing the reason will allow you to influence the business in the future and makes it possible to predict the results of an action.

Data analysis goals for the company

If we talk about business, then the goal of every company is to win the competition. So data analysis is your main advantage. It is he who will help you:

  • Reduce company expenses
  • Increase revenue
  • Reduce the time spent on the execution of business processes (find out the weak point and optimize it)
  • Increase the efficiency of the company's business processes
  • To fulfill any other goals aimed at improving the efficiency and effectiveness of the company.

This means that victory over competitors is in your hands. Don't rely on intuition. Analyze!

Data analysis goals for departments, divisions, products

Oddly enough, but the goals listed above are fully suitable for analyzing the activities of departments, analyzing a product or an advertising campaign.

The goal of any data analysis at any level is to identify patterns and use this knowledge to improve the quality of a product or the work of a company or department.

Who needs data analysis?

Everyone. Indeed, any company, from any field of activity, to any department and any product!

In what areas can data analysis be applied?

  • Manufacturing (construction, oil and gas, metallurgy, etc.)
  • Retail
  • Ecommerce
  • Services
  • And many others

Which departments can be analyzed within the company?

  • Accounting and finance
  • Marketing
  • Advertising
  • Administration
  • Other.

Indeed, companies from any sphere, any departments within the company, any areas of activity can, should and should be analyzed.

How BI Analysis Systems Can Help

BI analysis systems, automated systems analysts, big data for analyzing big data, are software solutions that already have built-in functionality for processing data, preparing them for analysis, analysis itself, and - most importantly - for visualizing the analysis results.

Not every company has an analyst department, or at least a developer, who will maintain the analytical system and databases. In this case, these BI-analysis systems come to the rescue.

There are more than 300 solutions on the market today. Our company settled on the Tableau solution:

  • In 2018, Tableau became the leader in BI solutions research by Gartner for the 6th time
  • Tableau is easy to learn (and our workshops prove it)
  • No developer knowledge or statistics required to fully get started with Tableau

At the same time, companies that already work with Tableau say that it now takes no more than 15 minutes to compile reports that were previously collected in Excel in 6-8 hours.

Don't believe me? Try it yourself - download a trial version of Tableau and get tutorials on how to use the program:

Download Tableau

Download for FREE full version Tableau Desktop 14 Days Get a GIFT of Tableau Business Intelligence Tutorials

Accessible work with Big Data using visual analytics

Improve business intelligence and solve routine tasks using information hidden in Big Data with the TIBCO Spotfire platform. It is the only platform that provides business users with an intuitive, easy-to-use user interface that enables the full range of analytic technologies for Big Data without the need for IT professionals or training.

The Spotfire interface makes it equally convenient to work with both small datasets and multi-terabyte clusters of big data: sensor readings, information from social networks, points of sale or geolocation sources. Users of all skill levels can easily navigate meaningful dashboards and analytic workflows simply by using visualizations that graphically represent the aggregation of billions of data points.

Predictive analytics is learning-by-doing based on shared experiences companies to make more reasoned decisions. Using Spotfire Predictive Analytics, you can discover new market trends from business intelligence insights and take action to minimize risk, leading to better management decisions.

Overview

Big Data Connectivity for High-Performance Analytics

Spotfire offers three main types of analytics with seamless integration with Hadoop and other large data sources:

  1. On-Demand Analytics Data Visualization: Built-in, user-configurable data connectors that facilitate ultra-fast, interactive data visualization
  2. Analysis in a database (In-Database Analytics): integration with a distribution computing platform that allows you to make calculations of data of any complexity based on big data.
  3. In-Memory Analytics: Platform Integration statistical analysis which takes data directly from any data source, including traditional and new data sources.

Together, these integration methods represent a powerful combination of visual exploration and advanced analytics.
It enables business users to access, aggregate, and analyze data from any data source through powerful, easy-to-use dashboards and workflows.

Big data connectors

Spotfire big data connectors support all kinds of data access: in-datasource, in-memory and on-demand. Spotfire's built-in data connectors include:

  • Hadoop Certified Data Connectors for Apache Hive, Apache Spark SQL, Cloudera Hive, Cloudera Impala, Databricks Cloud, Hortonworks, MapR Drill, and Pivotal HAWQ
  • Other certified big data connectors include Teradata, Teradata Aster, and Netezza
  • Connectors for historical and current data from sources such as OSI PI sensors

In-Datasource Distributed Computing

In addition to Spotfire's convenient visual selection of operations for SQL queries that access data distributed across sources, Spotfire can create statistical and machine learning algorithms that operate inside data sources and return only the results needed to create visualizations in Spotfire.

  • Users work with dashboards with visual selection functionality that access scripts using the built-in capabilities of the TERR language,
  • TERR scripts initiate distributed computing functionality in interaction with Map / Reduce, H2O, SparkR, or Fuzzy Logix,
  • These applications in turn access highly efficient systems like Hadoop or other data sources,
  • TERR can be deployed as an advanced analytics engine in Hadoop nodes that are driven by MapReduce or Spark. TERR can also be used for Teradata data nodes.
  • The results are visualized on Spotfire.

TERR for advanced analytics

TIBCO Enterprise Runtime for R (TERR) - TERR is a statistical package corporate level, which was developed by TIBCO for full compatibility with the R language, realizing the company's many years of experience in the analytical system associated with S +. This allows customers to continue to develop applications and models not only using open source R, but also integrate and deploy their R code on a commercial, reliable platform without having to rewrite their code. TERR has higher efficiency and reliable memory management, provides higher data processing speed on large volumes compared to the open source R language.

Combining all the functionality

Combining the aforementioned powerful functionality means that even for the most complex tasks requiring highly reliable analytics, users interact with simple, easy-to-use interactive workflows. This allows business users to visualize and analyze data, as well as share analytics results, without the need to know the details of the data architecture underlying the business analysis.

Example: Spotfire interface for configuring, running and visualizing the results of a model that defines the characteristics of lost loads. Through this interface, business users can perform computations using TERR and H2O (a distributed computing framework) by accessing transaction and shipment data stored in Hadoop clusters.

Analytical space for big data


Advanced and predictive analytics

Users use Spotfire's visual selection dashboards to launch a rich set of advanced features that make it easy to make predictions, create models, and optimize them on the fly. Using big data, analysis can be done inside the data source (In-Datasource), returning only the aggregated information and results needed to create visualizations on the Spotfire platform.


Machine learning

A wide range of machine learning tools are available in Spotfire's list of built-in features that can be used with a single click. Statisticians have access to the program code written in the R language and can expand the functionality used. Machine learning functionality can be shared with other users for easy reuse.

The following machine learning methods are available for continuous categorical variables on Spotfire and on TERR:

  • Linear and logistic regression
  • Decision trees, Random forest, Gradient Boosting Machine (GBM)
  • Generalized linear (additive) models ( Generalized Additive Models)
  • Neural networks


Content analysis

Spotfire provides analytics and data visualization, a significant part of which was not used before - it is unstructured text that is stored in sources such as documents, reports, notes CRM systems, site logs, publications in in social networks and much more.


Location analytics

Layered maps high resolution are a great way to visualize big data. Spotfire's rich map functionality allows you to create maps with as many reference and functional layers as you need. Spotfire also enables sophisticated analytics to be used while working with maps. In addition to geographic maps, the system creates maps to visualize the behavior of users, warehouses, production, raw materials and many other indicators.

Each big business and most middle-sized structures face the problem of providing management with inaccurate data on the state of affairs of the company. The reasons may be different, but the consequences are always the same - wrong or untimely decisions that negatively affect the effectiveness of financial transactions. To exclude such situations, a professional business analytics system or BI ( from English - Business Intelligence). These high-tech "helpers" help build the system management control every aspect within the business.

At its core, BI systems are advanced analytical software for business analysis and reporting. These programs can use data from various sources of information and provide them in a convenient form and cut. As a result, management gets quick access to complete and transparent information about the state of affairs of the company. A feature of reports obtained with the help of BI is the ability of the manager to independently choose in which context to receive information.


Modern Business Intelligence systems are multifunctional. That is why in large companies they are gradually replacing other methods of obtaining business reports. Experts refer to their main capabilities:

  • Connections to various databases, in particular, to;
  • Generation of reports of varying complexity, structure, type and layout with high speed... It is also possible to set a schedule for generating reports on a schedule without direct participation and distribution of data;
  • Transparent work with data;
  • Providing a clear connection between information from various sources;
  • Flexible and intuitive setting of employee access rights in the system;
  • Saving data in any format convenient for you - PDF, Excel, HTML and many others.

The capabilities of business intelligence information systems allow a manager not to depend on the IT department or his assistants to provide the required information. It is also a great opportunity to demonstrate the correct direction of your decisions, not in words, but in precise numbers. Many large network corporations in the West have been using BI systems for a long time, including the world famous Amazon, Yahoo, Wall-Mart and others. The above corporations spend a lot of money on business intelligence, but the implemented BI systems bring invaluable benefits.

The benefits of professional business intelligence systems are based on the principles that are supported in all advanced BI applications:

  1. Visibility. The main interface of any business analysis software should reflect the main indicators. Thanks to this, the manager will quickly be able to assess the state of affairs in the enterprise and begin to take something if necessary;
  2. Customization. Each user should be able to customize the interface and function keys in the most convenient way for themselves;
  3. Layering. Each dataset should have several sections (layers) to provide the level of detail that is needed at a particular level;
  4. Interactivity. Users should be able to collect information from all sources and from several directions at the same time. It is necessary that the system has the function of configuring the notification by key parameters;
  5. Multithreading and access control. In the BI system, the simultaneous operation of a large number of users should be implemented with the ability to set them different levels of access.

The entire IT community agrees that Information Systems business analysts are one of the most promising areas of industry development. However, their implementation is often hampered by technical and psychological barriers, uncoordinated work of managers and the absence of prescribed areas of responsibility.

When thinking about the implementation of BI-class systems, it is important to remember that the success of the project will largely depend on the attitude of the company's employees to innovation. This applies to all IT products: skepticism and fear of downsizing can undermine all implementation efforts. Therefore, it is very important to understand what feelings the business intelligence system evokes in future users. The ideal situation will arise when the company's employees treat the system as an assistant and a tool for improving their work.

Before starting a project for the implementation of BI technology, it is necessary to conduct a thorough analysis of the company's business processes and the principles of making management decisions. After all, it is these data that will participate in the analysis of the situation in the company. It will also help to make the choice of a BI system along with other main criteria:

  1. Goals and objectives of implementing BI systems;
  2. Requirements for storing data and the ability to operate with them;
  3. Data integration functions. Without using data from all sources in the company, management will not be able to get a holistic picture of the state of affairs;
  4. Visualization capabilities. For each person, the ideal BI analytics looks different, and the system must meet the needs of each user;
  5. Versatility or narrow specialization. There are systems in the world aimed at a specific industry, as well as universal solutions that allow you to collect information in any aspect;
  6. Demanding resources and the price of software... The choice of a BI system, like any software, depends on the capabilities of the company.

The above criteria will help the management make an informed choice among all the variety of well-known business intelligence systems. There are other parameters (eg storage structure, web architecture), but these require expertise in narrow IT areas.

It's not enough just to make a choice, buy software, install and configure it. Successful implementation of BI systems in any direction is based on the following rules:

  • Correctness of data. If the data for the analysis is incorrect, then there is a possibility of a serious system error;
  • Comprehensive training for each user;
  • Fast implementation. You need to focus on getting the right reports right at all key locations, rather than serving a single user perfectly. Adjust appearance report or add another section of it for convenience, you can always after implementation;
  • Realize the ROI on your BI system. The effect depends on many factors and in some cases is visible only after a few months;
  • The equipment should be designed not only for the current situation, but also for the near future;
  • Understand why the implementation of the BI system was started, and do not demand from software impossible.


According to statistics, only 30% of company executives are satisfied with the implementation of BI systems. Over the years of the existence of business analysis software, experts have formulated 9 key mistakes that can reduce efficiency to a minimum:

  1. Non-obviousness of the purpose of implementation for management. Often, a project is created by the IT department without the close involvement of managers. In most cases, in the process of implementation and operation, questions arise about the purpose and objectives of the BI system, the benefits and usability;
  2. Lack of transparency in management, work of employees and decision-making. Managers may not know the algorithms for the work of field employees, and management decisions can be accepted not only on the basis of dry facts. This will lead to the impossibility of maintaining the existing paradigm as a result of the implementation of the BI system. And often break the culture that has developed over the years corporate governance impossible;
  3. Insufficient data reliability. Falling false information into the business analysis system is unacceptable, otherwise employees will not be able to trust it and use it;
  4. The wrong choice of a professional business intelligence system. Many examples in history, when management hires a third-party organization to implement a BI system and does not take part in its choice, speak for themselves. As a result, a system is introduced that does not allow obtaining the required report or with which it is impossible to integrate one of the existing software in the company;
  5. Lack of a plan for the future. The peculiarity of BI systems is that it is not static software. It is impossible to finish an implementation project and not think about it. There are many requirements from users and management regarding improvements;
  6. Transfer of BI system outside organization for support. As practice shows, most often such situations lead to product isolation and isolation of the system from the real state of affairs. Own support service responds much faster and more efficiently to user feedback and management requirements;
  7. Desire to save money. In business, this is normal, but BI analytics only works if it takes into account all aspects of the company's activities. This is why high-value deep analytics systems are most effective. The desire to receive several reports on areas of interest leads to frequent data errors and a large dependence on the qualifications of IT specialists;
  8. Different terminology in the company. It is important that all users understand the basic terms and their meaning. A simple misunderstanding can lead to misinterpretation of the reports and indicators of the BI system;
  9. Lack of a unified business analysis strategy at the enterprise. Without a single course chosen for all employees, any BI class system will be just a set of disparate reports that satisfy the requirements of individual managers.

The implementation of BI systems is an important step that can help bring your business to the next level. But this will require not only a fairly large infusion of finance, but also the time and effort of each employee of the company. Not every business is ready to competently complete a project for implementing a business analysis system.


(Business Intelligence).

Young specialists are invited to the seminar as speakers. successful career analysts in high-tech companies such as Microsoft, IBM, Google, Yandex, MTS, etc. At each seminar, students are told about some of the business problems that are solved in these companies, how data is accumulated, how analysis problems arise data, by what methods they can be solved.

All invited specialists are open for contacts, and students will be able to contact them for advice.

Objectives of the workshop:

  • contribute to bridging the existing gap between university research and the solution of practical problems in the field of data analysis;
  • facilitate the exchange of experience between current and future professionals.
The seminar is held regularly at the Faculty of CMC MSU on Fridays at 18:20 , lecture hall P5(first floor).

Seminar attendance is free(if you do not have a pass to Moscow State University, please inform the organizers of the seminar in advance to submit the list of participants for the shift).

Workshop program

dateSpeaker and workshop topic
September 10, 2010
18:20
Alexander Efimov , head of analytical department retail network MTS.

Predicting the effect of marketing campaigns and optimizing store assortment.

  • Application page: Optimization of the assortment of outlets (data problem).
September 17, 2010
18:20
Vadim Strizhov , Researcher Computing Center RAS.

Bank Credit Scoring: Methods for Automatically Generating and Selecting Models.

The classic and new technology building scoring cards. The seminar explains how customer data works and how to generate the most plausible scoring model that meets, moreover, the requirements of international banking standards.

September 24, 2010
18:20
Vladimir Krekoten , Head of Marketing and Sales, Otkritie brokerage house.

Application mathematical methods to predict and counteract customer churn.

Practical problems arising in the analysis are considered. customer base in marketing. The tasks of clustering and segmentation of customers, scoring of new customers, tracking the dynamics of target segments are set.

  • Application page: Clustering clients of a brokerage company (data problem).
October 1, 2010
18:20
Nikolay Filipenkov , and about. Head of the Credit Scoring Department of the Bank of Moscow.

Applying Mathematical Methods to Manage Retail Credit Risk.

Some practical aspects of building scoring models and risk assessment are considered.

  • Application page: Retail Credit Risk Management (Data Problem).
October 8, 2010
18:20
Fedor Romanenko , Manager of the Search Quality Department, Yandex.

History and principles of web search ranking.

The article deals with the use and development of Information Retrieval methods, from text and link ranking to Machine Learning to Rank in the problem of Internet search. The fundamental principles behind modern web ranking are set out in relation to search engine success stories. Particular attention is paid to the impact of search quality on market indicators and the vital need to continually work to improve it.

October 15, 2010
18:20
Vitaly Goldstein , developer, Yandex.

Geographic information services Yandex.

It tells about the Yandex.Traffic jams project and other Yandex geoinformation projects, about where the initial data for building geoinformation systems come from, about a new scalable data processing technology, about the competition of Internet mathematics and some promising problems. Data is provided and a formal statement of the problem of road map restoration is given.

  • Application page: Building a road graph based on vehicle track data (data problem).
October 22, 2010The workshop was canceled.
October 29, 2010
18:20
Fedor Krasnov , VP of Business Processes and information technology, AKADO.

How do I get customer data?