Connect with us

AI

Drilling into Spark’s ALS Recommendation algorithm

Avatar

Published

on

The ALS algorithm introduced by Hu et al., is a very popular technique used in Recommender System problems, especially when we have implicit datasets (for example clicks, likes etc). It can handle large volumes of data reasonably well and we can find many good implementations in various Machine Learning frameworks. Spark includes the algorithm in the MLlib component which has recently been refactored to improve the readability and the architecture of the code.

Spark’s implementation requires the Item and User id to be numbers within integer range (either Integer type or Long within integer range), which is reasonable as this can help speed up the operations and reduce memory consumption. One thing I noticed though while reading the code is that those id columns are being casted into Doubles and then into Integers at the beginning of the fit/predict methods. This seems a bit hacky and I’ve seen it put unnecessary strain on the garbage collector. Here are the lines on the ALS code that cast the ids into doubles:

To understand why this is done, one needs to read the checkedCast():

This UDF receives a Double and checks its range and then casts it to integer. This UDF is used for Schema validation. The question is can we achieve this without using ugly double castings? I believe yes:

 protected val checkedCast = udf { (n: Any) => n match { case v: Int => v // Avoid unnecessary casting case v: Number => val intV = v.intValue() // True for Byte/Short, Long within the Int range and Double/Float with no fractional part. if (v.doubleValue == intV) { intV } else { throw new IllegalArgumentException(s"ALS only supports values in Integer range " + s"for columns ${$(userCol)} and ${$(itemCol)}. Value $n was out of Integer range.") } case _ => throw new IllegalArgumentException(s"ALS only supports values in Integer range " + s"for columns ${$(userCol)} and ${$(itemCol)}. Value $n is not numeric.") } }

The code above shows a modified checkedCast() which receives the input, checks asserts that the value is numeric and raises exceptions otherwise. Since the input is Any, we can safely remove all the cast to Double statements from the rest of the code. Moreover it is reasonable to expect that since the ALS requires ids within integer range, the majority of people actually use integer types. As a result on line 3 this method handles Integers explicitly to avoid doing any casting. For all other numeric values it checks whether the input is within integer range. This check happens on line 7.

One could write this differently and explicitly handle all the permitted types. Unfortunately this would lead to duplicate code. Instead what I do here is convert the number into Integer and compare it with the original Number. If the values are identical one of the following is true:

  1. The value is Byte or Short.
  2. The value is Long but within the Integer range.
  3. The value is Double or Float but without any fractional part.

To ensure that the code runs well I tested it with the standard unit-tests of Spark and manually by checking the behavior of the method for various legal and illegal values. To ensure that the solution is at least as fast as the original, I tested numerous times using the snippet below. This can be placed in the ALSSuite class in Spark:

 test("Speed difference") { val (training, test) = genExplicitTestData(numUsers = 200, numItems = 400, rank = 2, noiseStd = 0.01) val runs = 100 var totalTime = 0.0 println("Performing "+runs+" runs") for(i <- 0 until runs) { val t0 = System.currentTimeMillis testALS(training, test, maxIter = 1, rank = 2, regParam = 0.01, targetRMSE = 0.1) val secs = (System.currentTimeMillis - t0)/1000.0 println("Run "+i+" executed in "+secs+"s") totalTime += secs } println("AVG Execution Time: "+(totalTime/runs)+"s") }

After a few tests we can see that the new fix is slightly faster than the original:

Code

Number of Runs

Total Execution Time

Average Execution Time per Run

Original 100 588.458s 5.88458s
Fixed 100 566.722s 5.66722s

I repeated the experiments multiple times to confirm and the results are consistent. Here you can find the detailed output of one experiment for the original code and the fix. The difference is small for a tiny dataset but in the past I’ve managed to achieve a noticeable reduction in GC overhead using this fix. We can confirm this by running Spark locally and attaching a Java profiler on the Spark instance. I opened a ticket and a Pull-Request on the official Spark repo but because it is uncertain if it will be merged, I thought to share it here with you and it is now part of Spark 2.2.

Any thoughts, comments or critisism are welcome! 🙂

About Vasilis Vryniotis

My name is Vasilis Vryniotis. I’m a Data Scientist, a Software Engineer, author of Datumbox Machine Learning Framework and a proud geek. Learn more

Source: http://blog.datumbox.com/drilling-into-sparks-als-recommendation-algorithm/

AI

Cartoon: Cloud Dating

Avatar

Published

on

Cartoon: Cloud Dating

New KDnuggets cartoon looks at how AI can transform love and romance.


New KDnuggets cartoon gives you a respite from the virus and politics and issues of the day, and looks at how AI can transform love and romance.

Cartoon: Cloud Dating

A Scientist:

Our AI has come up with “Dating in the cloud”. It scans your social media posts and comes up with a great profile for you, automatically inflating your resume and making you more attractive. And no need to decie who pays for the meal!”

Here are other
KDnuggets AI, Big Data, Data Science, and Machine Learning Cartoons
.

and KDnuggets posts tagged
cartoon.

See also other recent KDnuggets Cartoons:

Source: https://www.kdnuggets.com/2020/10/cartoon-cloud-dating.html

Continue Reading

AI

How AI Revolutionize the Way Video Games Developed and Played

Avatar

Published

on

Artificial intelligence is simplifying our lives and changing the way we perform everyday tasks. An AI revolution has already taken place in many industries, like healthcare and marketing. But is it going to affect the future of video games as well? The simple answer is yes.

“Although AI has been an important part of video games since their birth in the ’50s, it’s only lately developed to a point where it can be used to control complex characters.” 

The technology can now process information from players and make the environment more suitable for their needs.  And what can we expect for video games in the future? Can we soon see self-thinking characters with complex countertactics in first-person shooter games? Or all the human staff replaced by robots in online websites with games like Eye of Horus? Let’s take a closer look at how exactly artificial intelligence will revolutionize the gaming industry. Ready?

Will Video Games Get Smarter?

To be honest, there hasn’t been a radical change in how AI governs the behaviour of various virtual entities or powers the procedural generation. AI’s most important components are still pathfinding and finite state machines.

Pathfinding is basically getting from one point to another in the simplest way, and it’s used in all games. And a finite state machine is built for non-playable characters (NPCs), and it allows them to change between different states.

However, some new video games are now also using behaviour and decision trees that have become more sophisticated recently. And although most developers are operating with the same basic concepts (pathfinding and finite-state), they can utilize them on bigger scales thanks to higher processing powers in 2020. 

Games Will Become More Adaptable to Smartphones

A gaming console is not the only equipment to play games on. A smartphone offers the unique possibility to play on the go, so it has become an increasingly popular option for gaming. And since the processing power of mobile phones grows every year, the games for these devices keep getting better and more immersive. 

However, video game fans still need a PC or a console to play the latest games since the weaker filling is not enough to enable mobile platforms to run them. But here’s where AI comes to play.

Every mobile game developer who creates new video games for smartphones is constantly coming up with new AI algorithms and the games keep getting smarter. Looking at mobile games only 3 years ago, we can see immense improvements. Let’s take a look at some key improvements in games made possible by advanced AI. 

Environments Are Getting More Authentic

Everyone’s probably noticed that video games are getting more realistic every year. The environments get more responsive and richer in detail. And that’s all thanks to AI. Using machine learning, AI can respond to any actions a player takes in the game more accurately. AI itself can generate environments that predict your expectations.

For example, the scripted plots that follow only one possible sequence of events are hardly present in any modern game. Instead, it’s up to the player’s decision on which direction the game takes. The plot has many possible directions. This means that you can play the game many times, and the scenario will always be fresh.

However, surprisingly enough, a powerful AI might not provide the most fun gaming experience. Because here’s the thing: if AI gets too unpredictable, the non-player characters it leads can start to direct the players to unnecessary routes. To places that lead to nowhere. It may be a fun experiment, but ultimately players are loving gaming because of the thrill of advancing to the next level. So most probably

AI in games will keep getting more intelligent to interact with players more convincingly. But it will still be completely controlled to keep everything on track. 

Improved Visuals 

The video graphics in modern games are so good that sometimes you can even mix your sense of what’s virtual and what’s reality. This has obviously not been achieved overnight. It’s made possible by hundreds of developers constantly working on new solutions. But we can hope AI to considerably accelerate the process by automating it.

Maybe in the future, we can have software that generates a game environment by scanning real cities. Just imagine having battles with zombies on the streets of your own hometown. 

Conclusion

In the future, we can expect more advanced AI in games making the non-player characters react to your actions with more variety. In first-person shooter games, NPCs can soon make more decisions and coordinate tactical attacks against the player. We can also expect more automized gaming sites. 

And AI is not only powering the characters and scenarios in games. It’s also put to work to develop games faster and cheaper than ever before. And that’s all revolutionary in the gaming business. Are you ready for it?

How do you think video games will look like in the future? Completely powered by AI? Leave comments below, we’d love to hear your thoughts.

                                                           — Thomas Glare endeavoured to explore the world of tech and is currently in his master’s program of Cyber Security who loves to create slot machine games for android.

Also, Read Inspiring AI Tech’s that Transformed Digital World

Source: https://www.aiiottalk.com/artificial-intelligence/how-ai-revolutionize-the-way-video-games-developed-and-played/

Continue Reading

AI

Internet of Things Impact in 2021

Avatar

Published

on

The internet of things — smart devices that connect to the internet to share and receive data — is probably one of the biggest tech success stories of the past 10 years. 

While most consumers know about IoT from consumer-oriented smart devices — like home assistants and security systems — the tech is also vital to many industries. From remote patient monitors to technology that collects factory data and controls massive industrial machinery, sending information over the internet has major benefits for all types of companies.

“The IoT revolution isn’t done yet. 2021 is likely to be another big year for it. We may soon see some of the biggest changes to date as developers find new ways to apply the tech to other industries.” 

Based on the IoT industry’s current momentum and direction, these eight changes are some of the most likely to happen next year.

1. New, More Interoperable Smart Home Devices

As the range of smart home devices grows, interoperability — having tools from different developers that can “talk” with each other — will become much more important.

Manufacturers of IoT devices are making communication between them a key design consideration. It may soon be more normal to coordinate devices from a handful of developers rather than depend on all-in-one solutions that manage your home environment or provide extra security. Instead, smart homeowners may be able to string together complex IoT tech systems, customizing based on their needs and preferences.

2. Adoption of IoT Will Be Almost Universal

According to a new report from Microsoft, 91% of global businesses have adopted IoT technology, up from 85% the year before. The same report showed that more than half of all companies plan to invest even more in IoT technology going forward.

The growing range of IoT solutions — from office management tech to new maintenance tools — means that any business can benefit from IoT tech regardless of industry. 

In 2021, it could be unusual to work with a company that doesn’t use the internet of things in one way or another. By the end of that year, we may see something close to 100% adoption of IoT.

3. 5G and Edge Computing Will Make Near-Real-Time Analysis Possible

After a slow rollout in 2019, major carriers are beginning to deliver 5G coverage on a national scale.

The speed of 5G, plus edge computing tech — which establishes nodes on the edge of the network that can be used for simple calculations — may make close-to-real time analytics in IoT devices possible. This ultra-fast communication will make the technology like driverless cars and real-time industrial sensors more practical going forward.

4. Hospitals Will Adopt a Range of IoT Tech

Over the past few years, developers have started to create various IoT devices for health care professionals. These devices — like internet-connected patient monitors or health-tracking wearables — make it easy for doctors and hospital staff to track a patient’s health, no matter where they are in the building. This could help staff check up on people across floors from one workstation or respond faster to a health crisis.

Shortly, real-time location systems for health care, which allow hospitals to track patients, manage assets and ensure compliance, may become central to the industry. At the least, it could become another valuable tool for health care professionals.

5. Cybersecurity Will Become a Serious Concern

As IoT devices proliferate, they’ll also become a bigger target for hackers. IoT security has improved significantly in the past few years, but they remain uniquely vulnerable to attacks. These safety risks make IoT devices a major opportunity for hackers wanting to break into a secure home or business network.

For developers and companies using IoT devices, keeping IoT tech secure will become critical. Investment in cybersecurity technology will likely grow to levels higher than anything we’ve seen so far.

6. Big-Name Retailers Will Implement Smart Store Technology

Following the semi-successful launch of Amazon’s line of cashier-less stores, Amazon Go, retailers have started to invest more in smart technology for their brick-and-mortar locations.

There are smart shelves, for example, outfitted with weight sensors and cameras to detect when products have run out. Similar tech can automatically check if there are carts that need to be pulled in or customers waiting to check out.

The data from these devices, coupled with AI-powered analytics, may make retail management more efficient. Right now, the tech is still experimental — but a few brands, like Walmart, are already testing smart store management at a few locations.

7. Predictive Maintenance Will Remain Key for Heavy Industry

Predictive maintenance is one of the more practical applications of the internet of things, and a decent number of industrial businesses already use it. The owner of heavy machinery or industrial infrastructure — like a robot arm, transformer or earth-mover — installs special sensors on certain pieces of high-value equipment. 

These sensors gather data on operational characteristics, like vibration, sound or temperature. Over time, this data analysis can give owners a heads-up when equipment is behaving unusually — a sign that it may need emergency maintenance or repairs.

The predictive maintenance market has grown fast over the past few years, and industry analysts estimate it will be worth more than $12 billion by 2025 — up from just $4 billion in 2020. 2021 will probably be a key year for the industry — one where we’ll see continued adoption of predictive maintenance tools and platforms.

8. Self-Driving Car Tests Will Ramp Up

More companies are getting involved in self-driving vehicles in one way or another. Ride-sharing companies, Big Tech businesses and logistics experts are all invested in making vehicles that can drive themselves.

Demand for self-driving cars, as well as other issues — like the ongoing shortage of truck drivers — will likely push companies to expand their tests in 2021. These vehicles still need some serious testing, and its likely tests won’t be done for a while. However, some of those experimental autonomous cars could be on the road by the end of 2021. 

How the IoT Could Reshape Businesses in 2021

The internet of things is likely to continue having a major influence on the economy through next year. New IoT devices, plus supporting techs like 5G and edge computing, mean that just about any business, regardless of industry, will be able to take advantage of what these tools can offer. 

Consumers are also likely to see big benefits as well. While self-driving cars aren’t expected to hit the market until later in the decade, other consumer IoT devices will probably become much more user-friendly. For smart home fanatics who want to outfit their living space with gadgets that can adjust the temperature, dim lights and keep an eye on the front porch, big changes to interoperability will make life better than ever.

Also, Read Future of IoT Devices

Source: https://www.aiiottalk.com/internet-of-things/impact-of-iot/

Continue Reading
Covid199 hours ago

Air Travel High: TSA Screens 1 Million For First Time Since March

Energy10 hours ago

CleanEquity® Monaco 2020 – Apresentando Empresas e Novas Colaborações

Energy10 hours ago

Steel Dynamics Reports Third Quarter 2020 Results

Energy11 hours ago

New Placer Dome Gold Corp to Webcast Live at VirtualInvestorConferences.com October 20th

Energy11 hours ago

EnLink Midstream Declares Third Quarter 2020 Distribution

Covid1911 hours ago

U.S. Borders With Canada And Mexico Will Stay Closed Another Month

Energy11 hours ago

Algonquin Completes ESSAL Acquisition

Energy13 hours ago

Global Force Sensors Markets to 2025: Improvement of Medical Devices with Force Sensor Technology will Drive the Market

Energy13 hours ago

Black Mamba Rod Lift and Oil Baron Supply Join Forces, Increasing Run-Times, Preventing Tubing Wear and Cavitation in Progressive Cavity Wells.

Esports13 hours ago

2K Games Alienates Players by Adding Unskippable Ads to NBA 2K21

Esports13 hours ago

Get Hype for Halloween With Hyper Scape’s Latest Event Trailer

Energy13 hours ago

Waterproofing Systems Market by Type, Application, and Region – Global Forecast to 2025

Esports13 hours ago

Rocket League Haunted Hallows Event Returns Oct. 20

Energy13 hours ago

$824 Million Worldwide Mobile Substation Industry to 2027 – Impact of COVID-19 on the Market

Esports13 hours ago

League of Legends Preseason 2021: 5 Things We Want

Esports13 hours ago

The Sims 4 Snowy Escape Pack Trailer Reveal is Coming Tuesday

Energy14 hours ago

Georgia Power launches new careers website for students as part of Careers in Energy Week

Cleantech15 hours ago

GM Unveils Factory ZERO

Covid1915 hours ago

UNICEF To Stockpile Over Half A Billion Syringes For Future COVID-19 Vaccine

Cleantech15 hours ago

Volvo Trucks Receives Grants to Deploy VNR Electric Trucks in Southern California

Covid1915 hours ago

South Korea Eases Coronavirus Restrictions, Touts ‘Exceptional’ Success

Crowdfunding16 hours ago

Rnwl: The First Over-the-Top Insurance Platform

Energy16 hours ago

Freeport-McMoRan’s Steve Higgins Elected as Chairman of the Board of the International Copper Association

Energy16 hours ago

Nufarm and CROP.ZONE Announce Cooperation to Bring Alternative Weed Control to Major European Markets

Energy17 hours ago

Global Belt and Chain Drives Market, 2020-2024: Growth Opportunities in Collaboration & Use of Newer Materials Enabling Broader Capabilities

Energy17 hours ago

New Report Shows Critical Impact of Oil and Gas Industry in Los Angeles County

Big Data17 hours ago

Best Apps to Check Internet Speed

France
Esports17 hours ago

Python joins Heretics

Energy17 hours ago

Ultra Safe Nuclear Technologies Delivers Advanced Nuclear Thermal Propulsion Design To NASA

AR/VR17 hours ago

The Virtual Arena: The Ascendance of Arena-Scale Entertainment – Part 1

Covid1918 hours ago

40 Million Coronavirus Cases Are Now Reported Worldwide

AR/VR18 hours ago

Pimax Secures $20m in Series B Funding Round

Fintech19 hours ago

Minimum Wage Workers Can Now Get Guaranteed Payday Loans No Matter What In Canada

Covid1919 hours ago

Top Palestinian Official Receiving COVID-19 Treatment In Israeli Hospital

Energy19 hours ago

Volvo Trucks Awarded $21.7M from U.S. EPA and South Coast AQMD to Deploy 70 Class 8 VNR Electric Zero-Emission Trucks

Energy19 hours ago

Trilliant Partners with 1NCE for a Cost-Effective Cellular Solution to Cover the Last Mile for IIoT

Energy19 hours ago

LyondellBasell Hosts Annual Global Care Day Supporting Food Security

Energy19 hours ago

Insider Buying Signals Gold Industry Momentum

Energy20 hours ago

In New Book, Veteran Journalist Shows How to End California’s Water Wars, Protect Habitats and Meet State’s Water Needs

Blockchain20 hours ago

How Does the Future Look for Cryptocurrencies in the Financial Market?

Trending