Data Platform to enhance Data Application experiences

By Vimal Venkatram, Country Head, India, Snowflake

As one of the largest and fastest-growing digital market, many companies in India are accelerating their technological adoption to maximize the value of their data. According to NASSCOM, cloud spending in India will reach US$7.2 billion by the year 2022, growing at a compound annual rate of 30 percent. The improvements in the digital ecosystem, increasing technological adoption of startup companies and opportunities within the small and medium-sized enterprises (SMEs) sector has pushed cloud platforms and big data companies to step up and match the market’s growing computing needs. With this rate of growth, India is projected to become a major supplier for Software as a Service (SaaS) products by the year 2022, contributing at least 15 percent to the total global demand.

Fast-growing SaaS and other cloud-based solutions providers have developed data applications for a variety of business use across markets, functions, and industries. Cloud platforms and big data companies are receiving huge volumes of valuable data that promises to unlock significant insights for their customers if managed properly. From marketing applications that provide customer insights, to Internet of Things (IoT) applications that handle device feedback, and data analytics applications that process both historical and near real-time data, the demand for data applications for analysis is growing exponentially.

Various data applications that promise to help companies take advantage of their data in real-time to improve business outcomes are emerging. However, many businesses using these applications struggle to extract and analyse these growing volumes of data efficiently. This is due to challenges that developers often face when building, designing, and supporting these applications including developing a 360-degree view of the customer data, handling IoT device data in near real-time, combining historical and current data for analysis, bringing data together for machine learning (ML) models, and embedding analytics in data-intensive applications.

A key obstacle in overcoming these challenges is application developers’ reliance on legacy architectures that only enable limited scalability, concurrency, and performance. To address these challenges, software companies can turn to the Data Cloud to build and manage their data stack. By adopting a modern, cloud-based data architecture, developers have an opportunity to deliver differentiated and defensible value to customers who need powerful features and real-time insights to run their businesses better.

Why does the architecture matter?
While applications have been modernised, the infrastructure powering these applications still runs on a traditional architecture that was built on the assumption that small clusters of machines with predictable amounts and types of structured data would be created largely by internal sources. Not surprisingly, these companies struggle with large volumes of data, as well as schema-less and semi-structured formats from external sources, such as application logs, web applications, mobile devices, social media, sensor data, and IoT data. This legacy architecture, created long before the emergence of the cloud, Infrastructure as a Service (IaaS) and Platform as a Service (PaaS), was not built to run massive SaaS applications with semi-structured data. Additionally, traditional data warehouses cannot scale to match data capacity or demand easily, which creates constraints on data availability. Adopting generic architectural plumbing and tools might be a quick and low-cost fix, but such strategy can cause technical challenges down the road that can lead to lower output and a disappointing customer experience.

As a result, investing in new architecture is critical to delivering on customer expectations of data applications. Some key capabilities to look for in a modern data platform for data applications are:
● Decoupled resources which allow applications to scale computing resources independently, and in a linear fashion for each job. It also enables multiple queries to be run against the same data without conflict.
● Elasticity to allow software companies to grow or shrink dynamically and adapt to load changes.
● Support for various data types to provide a holistic view of the data.
● Developer tooling and automation to enable developers to “plug in” services with application programming interface (API), applying a building blocks approach, rather than rebuilding with each addition to the application.
● Strong security baked into the design to enable fast development, while protecting against security threats.
To ensure that data applications deliver on their customers’ expectations, software companies need to align technology decisions with long-term product needs, keep evolving customer needs in mind, and design with growth and flexibility in mind. Application developers need a central repository to provide the workload isolation, instant and near-infinite elasticity, unlimited concurrency, and ability to natively ingest semi-structured data.
Additionally, many data application developers adopt generic low-cost tools that allow for quick development without upfront investment, as well as using traditional data platforms. When developers do not fully consider what is needed from their data stack to deliver powerful data analytics applications, problems can arise down the line. These problems include data storage and computing strains on the system, difficulties supporting semi-structured data, frequent maintenance and upgrades, and a lack of employee resources to configure the platform to their requirements. That will eventually lead to a full re-architecture of the data platform to address these issues, which can leave customers frustrated by latency issues and incomplete data analysis.
To develop powerful, responsive, and dynamic data applications, software companies need to invest in a modern, cloud-built data platform. Consider the data architecture before technical issues arise to enjoy a lower total cost of ownership from the beginning, remove the restraints of traditional data platforms and deliver fast, differentiated customer experiences.