Connect with us

Big Data

Analytics Engineering Everywhere

Published

on

Analytics Engineering Everywhere

Many new roles have appeared in the data world ever since the rise of the Data Scientist took the spotlight several years ago. Now, there is a new core player ready to take center stage, and we may see in five years, nearly every organization will have an Analytics Engineering team.


By Jason Ganz, a senior data analyst at GoCanvas.

Analytics Engineering — An Introduction

There’s a quiet revolution happening in the world of data. For years we have been blasted with nonstop articles about “The Sexiest Job of the 21st Century” — a data scientist. A data scientist, we have been taught, is a figure of almost otherworldly intelligence who uses quasi-mystical arts to perform feats of data wizardly. But these days, if you talk to the people who watch the data space most closely — there’s a different data role that has them even more excited.

To be clear, there are some very real and very cool applications of data science that can allow organizations to do things with data that can completely transform how their organization operates. But for many orgs, particularly smaller organizations without millions of dollars to invest, data science initiatives tend to fall flat because of the lack of a solid data infrastructure to support them.

While everyone was focused on the rise of data science, another discipline has been quietly taking shape, driven not by glitzy articles in Harvard Business Review but by the people working in the trenches in data-intensive roles. They call it the analytics engineer.

An analytics engineer is someone who brings together the data-savvy and domain knowledge of an analyst with software engineering tooling and best practices. Day to day, that means spending in a suite of tools that is becoming known as “The Modern Data Stack” and particularly dbt. These tools allow analytics engineers to centralize data and then model it for analysis in a way that is remarkably cheap and easy compared to how the ETL of traditional Business Intelligence teams of the past operated.

While data scientists are seen by some as wizardry, the attitude of the analytics engineer is a little different. You’ll hear them refer to themselves as everything from “humble data plumbers’’ to “just a pissed off data analyst.” The work of an Analytics Engineer seems easy to understand, almost banal. They combine data sources, apply logic, make sure there are clean and well-modeled materializations to analyze.

It turns out analytics engineering is a goddamn superpower. Anyone that has worked in, well, basically any organization knows that a tremendous amount of effort goes into standardizing data points that feel like they should be a no-brainer to pull, while more complex questions just sit unanswered for years. Analytics Engineering allows you to have data systems that just work.

A good analytics engineer is hugely impactful for an org, with each analytics engineer being able to help build a truly data-driven culture in ways that would be challenging for a team of people using legacy tools. While in the past there was tremendous repetitive work to do any simple analysis, Analytics Engineers can build complex data models using tools like dbt and have analysis-ready data tables built on any schedule. While before it was impossible to get anyone to agree on standard definitions of metrics, Analytics Engineers can simply build them into their codebase. And in the past, people struggled with incomplete and messy data, and Analytics Engineers… still struggle with incomplete and messy data. But at least we can have a suite of tests on our analytics systems to know when something has gone wrong!

The Rise of Analytics Engineering

You might think that this development would be scary for people working in data — if one analytics engineer is substantially more impactful than a data analyst, won’t our jobs be at risk? Could an org replace five data analysts with one Analytics Engineer and come out ahead?

But the fact of the matter is that no data analyst, anywhere, has ever come close to performing all of the analysis they think could be impactful at their organization — the opposite is far more likely to be the problem. Most data orgs are begging for additional headcount.

As analytics engineers increase the amount of insight organizations can find from data, it actually becomes more likely that these orgs will want to hire additional data talent (both analytics engineers and analysts). In his fantastic post The Reorganization of the Factory, Erik Bernhardsson makes the case that as the toolsets for software engineers has become ever more efficient, the demand for software engineers has counterintuitively grown — as there are more and more use cases where it now makes sense to build software rather than a manual process. This point not only holds for data, but I think it actually is more true for data.

While every organization needs software, not every organization needs software engineers. But every organization needs to learn from their data, and since the ways in which the data needs to be understood will be unique at every organization, they will all need analytics engineers. Software is commonly said to be eating the world — analytics engineering will be embedded in the world. As the incremental value of each data hire rises, there are substantial new areas where data insights and learnings could be applied that they aren’t today. And even if you aren’t interested in becoming an analytics engineer, having well modeled and accurate data makes data analysts and data scientists more effective. It’s a win across the board.

That does not necessarily mean that every analytics engineering role will be doing good for the world. Having more powerful data operations allows you to question, seek insights, and look for new strategies. It can also allow an organization new ways to monitor their employees, surveil, or discriminate. One needs only look at the myriad of public issues in the tech and data science industries right now to see the ways that powerful tech can be misused. It is important to recognize the potential dangers as well as the new opportunities.

If it feels like we’re at a real inflection point for Analytics Engineering — it’s because we are. What was very recently the domain of a few adventurous data teams is quickly becoming industry standard for tech organizations — and there’s every reason to think that other types of organizations will be following along shortly. The impact is just too high.

We’re about to see a huge expansion in the number of and types of places where you can find employment as an analytics engineer. The coming boom in opportunities for analytics engineers will take place across three rough domains, with each having different challenges and opportunities.

  • More and more large enterprises, both tech and non-tech organizations, are going to adapt to the modern data stack. As analytics engineering is brought into the most complex legacy data systems, we’ll begin to see what patterns develop to support analytics engineering at scale. If you are interested in really figuring out what the large-scale data systems of the future look like, this will be the place to go.
  • Just about every new company is going to be searching for an analytics engineer to lead their data initiatives. This will give them a step up against any competition that isn’t investing in their core data. Being an early analytics engineer at a fast-growing company is tremendously fun and exciting, as you are able to build up a data organization from scratch and see firsthand how analytics engineering can change the trajectory of an organization.
  • Finally, many organizations outside the tech business world are going to begin seeing the impact that analytics engineering can bring. You might not have quite the same tech budget, and you might have to learn to advocate for yourself a little more but it might be the area where analytics engineering has the most potential to do good for the world. City governments will use analytics engineering to monitor programs and ensure that government resources are being used effectively. Academic institutions will use analytics engineering to create datasets, many of them public, that will aid in scientific and technological development. The possibility space is wide open.

Analytics engineering is fundamentally a discipline that’s about making sense of the world around us. It’s about allowing everyone in an organization to see a little bit further in their impact on the org and how their work connects to it. Right now, analytics engineering is still a new discipline — pretty soon, it will be everywhere.

Original. Reposted with permission.

Related:

Coinsmart. Beste Bitcoin-Börse in Europa
Source: https://www.kdnuggets.com/2021/06/analytics-engineering-everywhere.html

Big Data

How much Mathematics do you need to know for Machine Learning?

Published

on



Mathematics For Machine Learning | Maths to understand ML Algorithms





















Learn everything about Analytics



PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://www.analyticsvidhya.com/blog/2021/07/how-much-mathematics-do-you-need-to-know-for-machine-learning/

Continue Reading

Big Data

If you did not already know

Published

on

ML Health google


Deployment of machine learning (ML) algorithms in production for extended periods of time has uncovered new challenges such as monitoring and management of real-time prediction quality of a model in the absence of labels. However, such tracking is imperative to prevent catastrophic business outcomes resulting from incorrect predictions. The scale of these deployments makes manual monitoring prohibitive, making automated techniques to track and raise alerts imperative. We present a framework, ML Health, for tracking potential drops in the predictive performance of ML models in the absence of labels. The framework employs diagnostic methods to generate alerts for further investigation. We develop one such method to monitor potential problems when production data patterns do not match training data distributions. We demonstrate that our method performs better than standard ‘distance metrics’, such as RMSE, KL-Divergence, and Wasserstein at detecting issues with mismatched data sets. Finally, we present a working system that incorporates the ML Health approach to monitor and manage ML deployments within a realistic full production ML lifecycle. …

Guided Zoom google


We propose Guided Zoom, an approach that utilizes spatial grounding to make more informed predictions. It does so by making sure the model has ‘the right reasons’ for a prediction, being defined as reasons that are coherent with those used to make similar correct decisions at training time. The reason/evidence upon which a deep neural network makes a prediction is defined to be the spatial grounding, in the pixel space, for a specific class conditional probability in the model output. Guided Zoom questions how reasonable the evidence used to make a prediction is. In state-of-the-art deep single-label classification models, the top-k (k = 2, 3, 4, …) accuracy is usually significantly higher than the top-1 accuracy. This is more evident in fine-grained datasets, where differences between classes are quite subtle. We show that Guided Zoom results in the refinement of a model’s classification accuracy on three finegrained classification datasets. We also explore the complementarity of different grounding techniques, by comparing their ensemble to an adversarial erasing approach that iteratively reveals the next most discriminative evidence. …

UniParse google


This paper describes the design and use of the graph-based parsing framework and toolkit UniParse, released as an open-source python software package. UniParse as a framework novelly streamlines research prototyping, development and evaluation of graph-based dependency parsing architectures. UniParse does this by enabling highly efficient, sufficiently independent, easily readable, and easily extensible implementations for all dependency parser components. We distribute the toolkit with ready-made configurations as re-implementations of all current state-of-the-art first-order graph-based parsers, including even more efficient Cython implementations of both encoders and decoders, as well as the required specialised loss functions. …

Sparse Constraint Preserving Matching (SPM) google


Many problems of interest in computer vision can be formulated as a problem of finding consistent correspondences between two feature sets. Feature correspondence (matching) problem with one-to-one mapping constraint is usually formulated as an Integral Quadratic Programming (IQP) problem with permutation (or orthogonal) constraint. Since it is NP-hard, relaxation models are required. One main challenge for optimizing IQP matching problem is how to incorporate the discrete one-to-one mapping (permutation) constraint in its quadratic objective optimization. In this paper, we present a new relaxation model, called Sparse Constraint Preserving Matching (SPM), for IQP matching problem. SPM is motivated by our observation that the discrete permutation constraint can be well encoded via a sparse constraint. Comparing with traditional relaxation models, SPM can incorporate the discrete one-to-one mapping constraint straightly via a sparse constraint and thus provides a tighter relaxation for original IQP matching problem. A simple yet effective update algorithm has been derived to solve the proposed SPM model. Experimental results on several feature matching tasks demonstrate the effectiveness and efficiency of SPM method. …

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://analytixon.com/2021/07/29/if-you-did-not-already-know-1461/

Continue Reading

Big Data

Nokia lifts full-year forecast as turnaround takes root

Published

on

HELSINKI (Reuters) -Telecom equipment maker Nokia reported a stronger-than-expected second-quarter operating profit on Thursday and raised its full-year outlook as promised, thanks to a turnaround of its business.

The Finnish company’s April-June comparable operating profit rose to 682 million euros ($808.51 million) from 423 million euros a year earlier, beating the 408-million euro mean estimate in a Refinitiv poll of analysts.

Shifting geopolitics and a sharp round of cost cutting have put Nokia firmly back in the global 5G rollout race just a year after CEO Pekka Lundmark took the reins, allowing it to gain ground on Swedish arch-rival Ericsson.

“We have executed faster than planned on our strategy in the first half which provides us with a good foundation for the full year,” Lundmark said in a statement on Thursday, but added that Nokia still expects the 2021 second-half results to be less pronounced.

Nokia said it now expects full-year net sales of 21.7 billion-22.7 billion euros, up from its prior estimate of 20.6 billion-21.8 billion euros, with an operating profit margin of 10-12% instead of the 7% to 10% expected previously.

The company had announced on July 13 that it would raise its outlook, but did not provide any details.

($1 = 0.8435 euros)

(Reporting by Essi Lehto; editing by Terje Solsvik and Sriraj Kaluvila)

Image Credit: Reuters

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://datafloq.com/read/nokia-lifts-full-year-forecast-turnaround-takes-root/16713

Continue Reading

Big Data

Robinhood, gateway to ‘meme’ stocks, raises $2.1 billion in IPO

Published

on

By Echo Wang and David French

(Reuters) -Robinhood Markets Inc, the owner of the trading app which emerged as the go-to destination for retail investors speculating on this year’s “meme’ stock trading frenzy, raised $2.1 billion in its initial public offering on Wednesday.

The company was seeking to capitalize on individual investors’ fascination with cryptocurrencies and stocks such as GameStop Corp, which have seen wild swings after becoming the subject of trading speculation on social media sites such as Reddit. Robinhood’s monthly active users surged from 11.7 million at the end of December to 21.3 million as of the end of June.

The IPO valued Robinhood at $31.8 billion, making it greater as a function of its revenue than many of its traditional rivals such as Charles Schwab Corp, but the offering priced at the bottom of the company’s indicated range.

Some investors stayed on the sidelines, citing concerns over the frothy valuation, the risk of regulators cracking down on Robinhood’s business, and even lingering anger with the company’s imposition of trading curbs when the meme stock trading frenzy flared up at the end of January.

Robinhood said it sold 55 million shares in the IPO at $38 apiece, the low end of its $38 to $42 price range. This makes it one of the most valuable U.S. companies to have gone public year-to-date, amid a red-hot market for new listings.

In an unusual move, Robinhood had said it would reserve between 20% and 35% of its shares for its users.

Robinhood’s platform allows users to make unlimited commission-free trades in stocks, exchange-traded funds, options and cryptocurrencies. Its simple interface made it popular with young investors trading from home during the COVID-19 pandemic.

Robinhood enraged some investors and U.S. lawmakers earlier this year when it restricted trading in some popular stocks following a 10-fold rise in deposit requirements at its clearinghouse. It has been at the center of many regulatory probes.

The company disclosed this week that it has received inquiries from U.S. regulators looking into whether its employees traded shares of GameStop and AMC Entertainment Holdings, Inc before the trading curbs were placed at the end of January.

In June, Robinhood agreed to pay nearly $70 million to settle an investigation by Wall Street’s own regulator, the Financial Industry Regulatory Authority, for “systemic” failures, including systems outages, providing “false or misleading” information, and weak options trading controls.

The brokerage has also been criticized for relying on “payment for order flow” for most of its revenue, under which it receives fees from market makers for routing trades to them and does not charge users for individual trades.

Critics argue the practice, which is used by many other brokers, creates a conflict of interest, on the grounds that it incentivizes brokers to send orders to whoever pays the higher fees. Robinhood contends that it routes trades based on what is cheapest for its users, and that charging a commission would be more expensive. The U.S. Securities and Exchange Commission is examining the practice.

Robinhood was founded in 2013 by Stanford University roommates Vlad Tenev and Baiju Bhatt. They will hold a majority of the voting power after the offering, these filings showed, with Bhatt having around 39% of the voting power of outstanding stock while Tenev will hold about 26.2%.

The company’s shares are scheduled to start trading on Nasdaq on Thursday under the ticker “HOOD”

Goldman Sachs and J.P. Morgan were the lead underwriters in Robinhood’s IPO.

(Reporting by Echo Wang and David French in New York; Editing by Leslie Adler)

Image Credit: Reuters

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://datafloq.com/read/robinhood-gateway-meme-stocks-raises-21-billion-ipo/16712

Continue Reading
Aviation4 days ago

Legendary F-14 Pilot Dale ‘Snort’ Snodgrass Dies In A Tragic Plane Crash

watch-live-russias-pirs-module-set-to-depart-space-station-today.jpg
Aerospace4 days ago

Watch live: Russia’s Pirs module set to depart space station today

Esports4 days ago

Genshin Impact Sacred Sakura Cleansing Ritual Quest Guide

Esports4 days ago

Genshin Impact Sacrificial Offering: How to Complete

Esports3 days ago

League of Legends Wild Rift Patch 2.4 Release Date

Esports5 days ago

Best Machamp build in Pokémon UNITE

Esports4 days ago

Where to Find Rivercress Stem in New World

Esports4 days ago

Best bot lane Pokémon on Pokémon UNITE

Energy4 days ago

NexGen Announces Commencement of 2021 Field and Regional Exploration Drilling Programs at the Rook I Property

Crowdfunding4 days ago

Digital Asset Firm Kraken Releases Report on Benefits of Centralized Finance Platforms Amid DeFi Boom

Esports5 days ago

Pokémon UNITE Crustle Build

Blockchain4 days ago

Ethereum 2.0 Exceeds 200K Validators, Has 6.6 Million ETH in Staking

Esports5 days ago

PUBG Mobile Global Championship (PMGC) 2021 unveiled with $6 million prize pool

Esports3 days ago

TFT Set 5.5 11.15 B-patch nerfs Hecarim, Lucian, and Irelia

AR/VR19 hours ago

Review: Winds & Leaves

Blockchain3 days ago

Ethereum Inventor Debuts As An Actor? Joins Mila Kunis In NFT-Based Show

Esports5 days ago

Best Wigglytuff build in Pokémon UNITE

Energy4 days ago

Nowa umowa partnerska Shanghai Electric zawarta podczas WAIC 2021 doprowadzi do rozwoju i przemiany wielu branż dzięki transformacji cyfrowej

Esports5 days ago

How to Beat Alastor the Vigilant in New World

Fintech3 days ago

Finding the right balance with hybrid client experiences

Trending